So I don't need to worry about billing issues now, right? I just need to pay a monthly subscription

WWang So I don't need to worry about billing issues now, right? I just need to pay a m...

U

Unsmart•2/19/24, 3:09 AM

Yes. You can see what the pricing will be here though: https://developers.cloudflare.com/vectorize/platform/pricing/

W

WangOP•2/19/24, 3:10 AM

Thanks

W

WangOP•2/19/24, 3:10 AM

When billing starts according to the document, there will be a notification, right?

U

Unsmart•2/19/24, 3:10 AM

Yes there will be

W

WangOP•2/19/24, 3:13 AM

One more thing, are there more API documents? I want to know what data I have stored, along with their metadata information, in order to provide some visualizations on the pages for customers to browse and update.

W

WangOP•2/19/24, 3:13 AM

The document currently available only includes Create indexes, Insert vectors, Query vectors.

U

Unsmart•2/19/24, 3:14 AM

All the api docs are here: https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes

U

Unsmart•2/19/24, 3:14 AM

Or here: https://developers.cloudflare.com/vectorize/reference/client-api/

W

WangOP•2/19/24, 3:14 AM

Thanks

L

leesus•2/20/24, 1:01 PM

Is vector retrieval using

getByIds

getByIds

getByIds

getByIds classed as a query with regards to billing?

MMatt Silverlock Soon! Ideally before the next holiday period. Prioritizing some stability/scalab...

K

kingmesal•2/20/24, 5:38 PM

Any updates on when vectorize bindings will be available to pages projects?

A

anthony•2/21/24, 6:32 AM

Also regarding vector retrieval using

getByIds

getByIds

getByIds

getByIds, i believe the max number of Ids is

20

20 from some testing but I can't find the documentation for it, will this limit be increased?

O

Old Man Maker•2/23/24, 8:35 AM

Hey folks - I'm not understanding an issue that I am seeing with Filtering

Here is an object returned with no filtering:

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

Please note the prjRef value in the metadata.

When I now call the index exactly the same but this time with filtering, I get zero returned vectors in count and or matches:

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

O

Old Man Maker•2/23/24, 8:38 AM

I'm thinking this is because of the way I am creating metedata in the first place?

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

In my mind, the filtering would be trying to filter on the "data" value without any access to the JSON object held in it?

O

Old Man Maker•2/23/24, 9:13 AM

And yes... my earliest Vectorize code was at fault here. Having pushed the metadata directly into the vector, filtering is working as expected!

DDhravya Shah hey, i've been trying to add article-size embeddings to vectorize, which, predic...

I

Isaac McFadyen•2/28/24, 7:16 PM

When you create the vectorize index you can specify the vector size I believe.

I

Isaac McFadyen•2/28/24, 7:16 PM

There's presets but you can also hit custom and provide your own size.

I

Isaac McFadyen•2/28/24, 7:17 PM

Just note that I believe they expect all vectors to be the same size.

G

guanhuigithub•2/28/24, 8:18 PM

hi

W

WangOP•2/29/24, 11:11 AM

I use

env.VECTORIZE_INDEX.upsert(vectors);

env.VECTORIZE_INDEX.upsert(vectors);

env.VECTORIZE_INDEX.upsert(vectors);

env.VECTORIZE_INDEX.upsert(vectors);

and it takes less than 200ms. However, when I upsert vectors using the REST API, it takes 88856ms. Is this normal?

W

WangOP•2/29/24, 11:13 AM

This example

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

R

Rasel Rana Rocky•3/6/24, 10:48 AM

Couple questions:
I need vectorize for production yesterday

I don't want to deal with pgvector or anything alike.

Could it be possible to get the maximum vectors per index increased?
Can I get more namespace?
Is there gonna be any breaking changes? I hope not since this is public beta.

We already have a live product. We REALLY need to introduce GPT into the support assistant system. So, we have A LOT OF users from day one.

RRasel Rana Rocky Couple questions: I need vectorize for production yesterday 😀 I don't want to ...

U

Unsmart•3/6/24, 3:22 PM

Vectorize is a beta product thus many of the limits are something that will eventually increase but I would assume you probably cannot really request increases atm. Generally cloudflare tries to not have breaking changes but there has been 1 for vectorize iirc

UUnsmart Vectorize is a beta product thus many of the limits are something that will even...

O

Old Man Maker•3/7/24, 7:55 AM

Correct - there was a breaking change for vector metadata filtering

A

Angel Luis•3/8/24, 3:19 PM

It would be nice to be able to see the vectors inside the Vectorize dashboard like other vector databases like Qdrant

A

Angel Luis•3/8/24, 3:20 PM

or at least from Wrangler, now I'm working blindly, I've to make my own script to get the vectors through the Worker

A

Angel Luis•3/8/24, 4:03 PM

to be honest I've been working 2 hours with Vectorize and it's a disaster. It's like to have a database without access to any row, the only way to have any track is creating your own fetcher with

getByIds()

getByIds()

getByIds()

getByIds(), track exactly what ids you have inserted, and at maximum fetch 20 ids.

A

Angel Luis•3/8/24, 4:04 PM

I've been working with Milvus and Qdrant, and Vectorize is still very far

A

Angel Luis•3/8/24, 4:28 PM

also I've passed by mistake literally 20 times

[ Object, Object ]

[ Object, Object ]

[ Object, Object ]

[ Object, Object ] as id to a vector index, so the only way I've to fix this is removing the entire vector

omg

A

Angel Luis•3/8/24, 4:45 PM

I will share my current script to help others, but in my opinion the service is too experimental to be public...

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

(and this is a tremendous waste of resources...)

A

Angel Luis•3/11/24, 2:36 PM

I've been thinking and in my opinion Vectorize should have its own numeric index as id, like any SQL database, and then with metadata we can skip the need of using a D1 database, like other vector databases systems. Then just adding a simple way to delete vectors via CLI based on filtering would do the trick. But at this point, the only purpose I'm using D1 is for tracking the ID and don't work blindly. With that I mean that for example, I've a Vector index called "image generation papers", and let's say I've the paper A, which occupies 30 vector entries. Then, if paper A needs and update, the only way I've to track it is via D1, fetching its ids based on the column I've provided in the moment of the vectorization

K

Korcan•3/11/24, 7:57 PM

Hello, is it possible to use Vectorize with Cloudflare Pages?

W

WangOP•3/12/24, 6:44 AM

@Korcan Sure, you can use the REST API for invocation.

K

knoedelkopp•3/12/24, 2:51 PM

Hi!

I was wondering: are vector IDs unique per index or unique per namespace?

Reading https://developers.cloudflare.com/vectorize/reference/client-api/#vectors , I think it's unique per index?

Cloudflare Docs

Vectorize API · Vectorize

This page covers the Vectorize API available within Cloudflare Workers, including usage examples.

A

A-Social-Ninja-University•3/16/24, 9:14 AM

ninja-mann07.github.ioi

Ccarrot sticks how silly will it be for me to build my website around this vectorize functional...

D

dakom•3/18/24, 11:43 AM

any n00b tips to learn more about how to use vector search for that kinda thing? still getting my feet wet in this area coming from traditional dbs

Kknoedelkopp Hi! I was wondering: **are vector IDs unique per index or unique per namespace?...

J

Jerome•3/22/24, 1:43 PM

yep, unique per index

O

Old Man Maker•3/22/24, 3:39 PM

Hi - can someone check a vector index for me please. I am looking at some logs that show an insert call but the return is :

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

{
      "message": [
        "Response from Index =>",
        "{\"count\":0,\"ids\":[]}"
      ],
      "level": "log",
      "timestamp": 1711117766889
    },

No other information why the supplied values are not being inserted
What information is needed from me to check this?

V

Victor•3/25/24, 8:30 AM

Hey there, I was wondering if there was any benefit to using namespaces with regards to billing?
Say you have a namespace with a large number of vectors, and a smaller one, it would still be preferable to create separate indexes to reduce the query cost, right?

KKorcan Hello, is it possible to use Vectorize with Cloudflare Pages?

S

schkovich•3/25/24, 11:15 PM

To overcome the lack of Vectorize in Pages, I use Service Bindings as a reverse proxy to the Vectorize database.

In the application code (functions), I have a class that acts as a facade, implementing all the methods from the Vectorize bindings that are, in the background, sending requests to the bound worker, which would proxy the request to the Vectorize database and the response back.

There are downsides to such an approach. The most notable one is that only remote development is supported.

P

pkpio•3/26/24, 2:05 PM

Is topK expected to bump soon? Realistically what value can we expect?

A

Anubhav Singh•3/31/24, 12:28 PM

I have tried almost everything trying to get the

upsert

upsert

upsert

upsert or

insert

insert

insert

insert method to work for me via the REST API or the

cloudflare

cloudflare

cloudflare

cloudflare npm package but keep getting

  errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

  errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

errors: [ { code: 1005, message: 'vectorize.unknown_content_type' } ]

The same code when used via wrangler (and bindings) works flawlessly.

Can anyone please share a working sample of upsert/insert via REST API?

Ccfnathan Are you sure you're specifying the content type (`application/x-ndjson`) when se...

A

Anubhav Singh•4/1/24, 6:02 AM

All the samples on the API docs too want me to submit ndjson files, but I do not want to involve files in the process. If its possible to do this via wrangler's vector bindings, whose code I extracted and tried using without success, it should be possible to use via REST API too!

Ccfnathan We're in the midst of getting things in ship shape for a GA release of Vectorize...

P

pkpio•4/3/24, 9:51 AM

Hoping to see 200. Since LLMs context is expanding, for certain queries we can provide better answers by including more context. For example, tasks like summarize a video transcript. The current 25 docs limit is simply not usable to produce a good enough summary for lack of sufficient information in the context.

Ccfnathan We're in the midst of getting things in ship shape for a GA release of Vectorize...

A

AndyJessop•4/4/24, 8:27 AM

At the current time, 100 would work for my purposes. But with context windows going up exponentially, it's making VectorDB results less and less useful. At some point I'll be able to fit my entire repo into context and that would eclipse any benefit from Vectors. In that light, I think that topK needs to start high and go higher (maybe in proportion to context window size?).

So I don't need to worry about billing issues now, right? I just need to pay a monthly subscription

Similar Threads

Similar Threads

Similar Threads