Why is in this calculation the number of vectors added to the number of queries. Wouldn't it make mo

T

Tarcísio•2/8/24, 7:54 PM

Holla

YYanni Why is in this calculation the number of vectors added to the number of queries....

Y

YanniOP•2/8/24, 7:58 PM

does this 10000 only count for the first time (when you insert it in the vectorstore)? And every query after that against the vectorstore just costs the number of vector dimensions?

YYanni does this 10000 only count for the first time (when you insert it in the vectors...

V

Vy•2/9/24, 8:00 PM

yes, its following the formula in https://developers.cloudflare.com/vectorize/platform/pricing/#calculating-vector-dimensions with

(stored vectors + queried vectors) * dimensions

(stored vectors + queried vectors) * dimensions

(stored vectors + queried vectors) * dimensions

(stored vectors + queried vectors) * dimensions

Cloudflare Docs

Pricing · Vectorize

Vectorize bills based on:

VVy yes, its following the formula in https://developers.cloudflare.com/vectorize/pl...

Y

YanniOP•2/10/24, 11:44 AM

Wouldnt it make sense to expicit mention this in the documentation. Since it is a little confusing

Y

YanniOP•2/10/24, 11:47 AM

Finding the most similar vector requires calculating the cosinus. Which requires more computing power when more vectores are stored in the database. Yes there are ways to optimize i. But the first thing that comes to my mind when reading through documentation is that the number of queries is the times that 2 vectores got compared.

W

Wang•2/19/24, 2:29 AM

Hello, Cloudflare staff.

W

Wang•2/19/24, 2:29 AM

Want to know how to get vectorize some stored information, such as I want to create a web page showing the vectors and metadata that I have already stored.

W

Wang•2/19/24, 2:30 AM

The document currently available only includes Create indexes, Insert vectors, Query vectors.

W

Wang•2/19/24, 3:05 AM

And where can I view the vectorize billing page?

W

Wang•2/19/24, 3:05 AM

Thanks

U

Unsmart•2/19/24, 3:05 AM

There is no billing page because its not billed currently

U

Unsmart•2/19/24, 3:06 AM

fwiw though theres not really a billing page for most things just analytics in the upper right when you go to the product as far as workers products

W

Wang•2/19/24, 3:06 AM

UUnsmart There is no billing page because its not billed currently

W

Wang•2/19/24, 3:08 AM

So I don't need to worry about billing issues now, right? I just need to pay a monthly subscription fee of $5.

WWang So I don't need to worry about billing issues now, right? I just need to pay a m...

U

Unsmart•2/19/24, 3:09 AM

Yes. You can see what the pricing will be here though: https://developers.cloudflare.com/vectorize/platform/pricing/

W

Wang•2/19/24, 3:10 AM

Thanks

W

Wang•2/19/24, 3:10 AM

When billing starts according to the document, there will be a notification, right?

U

Unsmart•2/19/24, 3:10 AM

Yes there will be

W

Wang•2/19/24, 3:13 AM

One more thing, are there more API documents? I want to know what data I have stored, along with their metadata information, in order to provide some visualizations on the pages for customers to browse and update.

W

Wang•2/19/24, 3:13 AM

The document currently available only includes Create indexes, Insert vectors, Query vectors.

U

Unsmart•2/19/24, 3:14 AM

All the api docs are here: https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes

U

Unsmart•2/19/24, 3:14 AM

Or here: https://developers.cloudflare.com/vectorize/reference/client-api/

W

Wang•2/19/24, 3:14 AM

Thanks

L

leesus•2/20/24, 1:01 PM

Is vector retrieval using

getByIds

getByIds

getByIds

getByIds classed as a query with regards to billing?

MMatt Silverlock Soon! Ideally before the next holiday period. Prioritizing some stability/scalab...

K

kingmesal•2/20/24, 5:38 PM

Any updates on when vectorize bindings will be available to pages projects?

A

anthony•2/21/24, 6:32 AM

Also regarding vector retrieval using

getByIds

getByIds

getByIds

getByIds, i believe the max number of Ids is

from some testing but I can't find the documentation for it, will this limit be increased?

O

Old Man Maker•2/23/24, 8:35 AM

Hey folks - I'm not understanding an issue that I am seeing with Filtering

Here is an object returned with no filtering:

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

Please note the prjRef value in the metadata.

When I now call the index exactly the same but this time with filtering, I get zero returned vectors in count and or matches:

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

O

Old Man Maker•2/23/24, 8:38 AM

I'm thinking this is because of the way I am creating metedata in the first place?

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

In my mind, the filtering would be trying to filter on the "data" value without any access to the JSON object held in it?

O

Old Man Maker•2/23/24, 9:13 AM

And yes... my earliest Vectorize code was at fault here. Having pushed the metadata directly into the vector, filtering is working as expected!

DDhravya Shah hey, i've been trying to add article-size embeddings to vectorize, which, predic...

I

Isaac McFadyen•2/28/24, 7:16 PM

When you create the vectorize index you can specify the vector size I believe.

I

Isaac McFadyen•2/28/24, 7:16 PM

There's presets but you can also hit custom and provide your own size.

I

Isaac McFadyen•2/28/24, 7:17 PM

Just note that I believe they expect all vectors to be the same size.

G

guanhuigithub•2/28/24, 8:18 PM

hi

W

Wang•2/29/24, 11:11 AM

I use

env.VECTORIZE_INDEX.upsert(vectors);

env.VECTORIZE_INDEX.upsert(vectors);

env.VECTORIZE_INDEX.upsert(vectors);

env.VECTORIZE_INDEX.upsert(vectors);

and it takes less than 200ms. However, when I upsert vectors using the REST API, it takes 88856ms. Is this normal?

W

Wang•2/29/24, 11:13 AM

This example

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

curl --request POST \
  --url https://api.cloudflare.com/client/v4/accounts/account_identifier/vectorize/indexes/index_name/upsert \
  --header 'Authorization: Bearer undefined' \
  --header 'Content-Type: application/x-ndjson' \
  --data @/path/to/vectors.ndjson

R

Rasel Rana Rocky•3/6/24, 10:48 AM

Couple questions:
I need vectorize for production yesterday

I don't want to deal with pgvector or anything alike.

Could it be possible to get the maximum vectors per index increased?
Can I get more namespace?
Is there gonna be any breaking changes? I hope not since this is public beta.

We already have a live product. We REALLY need to introduce GPT into the support assistant system. So, we have A LOT OF users from day one.

RRasel Rana Rocky Couple questions: I need vectorize for production yesterday 😀 I don't want to ...

U

Unsmart•3/6/24, 3:22 PM

Vectorize is a beta product thus many of the limits are something that will eventually increase but I would assume you probably cannot really request increases atm. Generally cloudflare tries to not have breaking changes but there has been 1 for vectorize iirc

UUnsmart Vectorize is a beta product thus many of the limits are something that will even...

O

Old Man Maker•3/7/24, 7:55 AM

Correct - there was a breaking change for vector metadata filtering

A

Angel Luis•3/8/24, 3:19 PM

It would be nice to be able to see the vectors inside the Vectorize dashboard like other vector databases like Qdrant

A

Angel Luis•3/8/24, 3:20 PM

or at least from Wrangler, now I'm working blindly, I've to make my own script to get the vectors through the Worker

A

Angel Luis•3/8/24, 4:03 PM

to be honest I've been working 2 hours with Vectorize and it's a disaster. It's like to have a database without access to any row, the only way to have any track is creating your own fetcher with

getByIds()

getByIds()

getByIds()

getByIds(), track exactly what ids you have inserted, and at maximum fetch 20 ids.

A

Angel Luis•3/8/24, 4:04 PM

I've been working with Milvus and Qdrant, and Vectorize is still very far

A

Angel Luis•3/8/24, 4:28 PM

also I've passed by mistake literally 20 times

[ Object, Object ]

[ Object, Object ]

[ Object, Object ]

[ Object, Object ] as id to a vector index, so the only way I've to fix this is removing the entire vector

omg

A

Angel Luis•3/8/24, 4:45 PM

I will share my current script to help others, but in my opinion the service is too experimental to be public...

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

const limitChunk = 5; // 20 * 5 = 100 ids
const results = [];

for (let cChunck = 1; cChunck <= limitChunk; cChunck++) {
  const ids = Array.from({ length: 20 }, (_, i) =>
      (20 * (cChunck - 1) + (i + 1)).toString(),
  );
  const result = await env.VECTORIZE_INDEX.getByIds(ids);
  results.push(result);
}

const vectors = results.map((arr) => arr.map(({ values, ...rest }) => rest));

(and this is a tremendous waste of resources...)

A

Angel Luis•3/11/24, 2:36 PM

I've been thinking and in my opinion Vectorize should have its own numeric index as id, like any SQL database, and then with metadata we can skip the need of using a D1 database, like other vector databases systems. Then just adding a simple way to delete vectors via CLI based on filtering would do the trick. But at this point, the only purpose I'm using D1 is for tracking the ID and don't work blindly. With that I mean that for example, I've a Vector index called "image generation papers", and let's say I've the paper A, which occupies 30 vector entries. Then, if paper A needs and update, the only way I've to track it is via D1, fetching its ids based on the column I've provided in the moment of the vectorization

K

Korcan•3/11/24, 7:57 PM

Hello, is it possible to use Vectorize with Cloudflare Pages?

W

Wang•3/12/24, 6:44 AM

@Korcan Sure, you can use the REST API for invocation.

Why is in this calculation the number of vectors added to the number of queries. Wouldn't it make mo

Similar Threads

Similar Threads

Similar Threads