how silly will it be for me to build my website around this vectorize functionality? 😄 i plan on u

C

carrot sticksOP•1/25/24, 4:13 PM

Thank you nathan, never saw that. Peanuts cost as usual

Ccfnathan Ah, sorry, our release yesterday undid the change we made: let me make the chang...

D

divby0•1/25/24, 4:29 PM

can you update here when you did deploy the change permanent? Or did you already (cause I am still not able to get a OK status code)

D

divby0•1/25/24, 4:44 PM

awesome works!

Ccfnathan This should be in place now - if you still see 502's let us know 🙂

D

divby0•1/25/24, 4:44 PM

Thanks you

Y

yevgen•2/1/24, 1:41 PM

Hi @Anders we've made a decision to lockdown any changes to our APIs until we ship multi-million vectors support per index. We will review future API changes after that.

W

Wang•2/6/24, 7:10 AM

Hi, Why can't I find the "vectoize" function in the menu bar? Thank you.

W

Wang•2/6/24, 7:10 AM

C

chientrm.com•2/8/24, 4:09 AM

I don't see

Vectorize

Vectorize

Vectorize

Vectorize too

Y

Yanni•2/8/24, 7:53 PM

Why is in this calculation the number of vectors added to the number of queries. Wouldn't it make more sense to multiply them?

T

Tarcísio•2/8/24, 7:54 PM

Holla

YYanni Why is in this calculation the number of vectors added to the number of queries....

Y

Yanni•2/8/24, 7:58 PM

does this 10000 only count for the first time (when you insert it in the vectorstore)? And every query after that against the vectorstore just costs the number of vector dimensions?

YYanni does this 10000 only count for the first time (when you insert it in the vectors...

V

Vy•2/9/24, 8:00 PM

yes, its following the formula in https://developers.cloudflare.com/vectorize/platform/pricing/#calculating-vector-dimensions with

(stored vectors + queried vectors) * dimensions

(stored vectors + queried vectors) * dimensions

(stored vectors + queried vectors) * dimensions

(stored vectors + queried vectors) * dimensions

Cloudflare Docs

Pricing · Vectorize

Vectorize bills based on:

VVy yes, its following the formula in https://developers.cloudflare.com/vectorize/pl...

Y

Yanni•2/10/24, 11:44 AM

Wouldnt it make sense to expicit mention this in the documentation. Since it is a little confusing

Y

Yanni•2/10/24, 11:47 AM

Finding the most similar vector requires calculating the cosinus. Which requires more computing power when more vectores are stored in the database. Yes there are ways to optimize i. But the first thing that comes to my mind when reading through documentation is that the number of queries is the times that 2 vectores got compared.

W

Wang•2/19/24, 2:29 AM

Hello, Cloudflare staff.

W

Wang•2/19/24, 2:29 AM

Want to know how to get vectorize some stored information, such as I want to create a web page showing the vectors and metadata that I have already stored.

W

Wang•2/19/24, 2:30 AM

The document currently available only includes Create indexes, Insert vectors, Query vectors.

W

Wang•2/19/24, 3:05 AM

And where can I view the vectorize billing page?

W

Wang•2/19/24, 3:05 AM

Thanks

U

Unsmart•2/19/24, 3:05 AM

There is no billing page because its not billed currently

U

Unsmart•2/19/24, 3:06 AM

fwiw though theres not really a billing page for most things just analytics in the upper right when you go to the product as far as workers products

W

Wang•2/19/24, 3:06 AM

UUnsmart There is no billing page because its not billed currently

W

Wang•2/19/24, 3:08 AM

So I don't need to worry about billing issues now, right? I just need to pay a monthly subscription fee of $5.

WWang So I don't need to worry about billing issues now, right? I just need to pay a m...

U

Unsmart•2/19/24, 3:09 AM

Yes. You can see what the pricing will be here though: https://developers.cloudflare.com/vectorize/platform/pricing/

W

Wang•2/19/24, 3:10 AM

Thanks

W

Wang•2/19/24, 3:10 AM

When billing starts according to the document, there will be a notification, right?

U

Unsmart•2/19/24, 3:10 AM

Yes there will be

W

Wang•2/19/24, 3:13 AM

One more thing, are there more API documents? I want to know what data I have stored, along with their metadata information, in order to provide some visualizations on the pages for customers to browse and update.

W

Wang•2/19/24, 3:13 AM

The document currently available only includes Create indexes, Insert vectors, Query vectors.

U

Unsmart•2/19/24, 3:14 AM

All the api docs are here: https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes

U

Unsmart•2/19/24, 3:14 AM

Or here: https://developers.cloudflare.com/vectorize/reference/client-api/

W

Wang•2/19/24, 3:14 AM

Thanks

L

leesus•2/20/24, 1:01 PM

Is vector retrieval using

getByIds

getByIds

getByIds

getByIds classed as a query with regards to billing?

MMatt Silverlock Soon! Ideally before the next holiday period. Prioritizing some stability/scalab...

K

kingmesal•2/20/24, 5:38 PM

Any updates on when vectorize bindings will be available to pages projects?

A

anthony•2/21/24, 6:32 AM

Also regarding vector retrieval using

getByIds

getByIds

getByIds

getByIds, i believe the max number of Ids is

20

20 from some testing but I can't find the documentation for it, will this limit be increased?

O

Old Man Maker•2/23/24, 8:35 AM

Hey folks - I'm not understanding an issue that I am seeing with Filtering

Here is an object returned with no filtering:

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

{
    \"count\":1,
    \"matches\":[
        {
            \"id\":\"5b449757-7d4f-44ee-af26-05301ea5384c\",
            \"namespace\":\"content_items\",
            \"values\":[
                0.025468263775110245,
                <--THE REST OF THE VECTORS-->,
                0.015441170893609524],
            \"metadata\":{
                \"data\":\"{
                    \\\"d1Key\\\":\\\"5b449757-7d4f-44ee-af26-05301ea5384c\\\",
                    \\\"prjRef\\\":\\\"dd6f0ae0-68e6-4651-8578-8a4307e4612b\\\",
                    \\\"fltRef\\\":\\\"1c5d07ce-189a-4a97-8066-d2c72aaa7494\\\",
                    \\\"orgRef\\\":\\\"38a3861b-573a-43fc-94ed-4ad8d639ff80\\\",
                    \\\"scope\\\":1,
                    \\\"vecType\\\":1
                    }\"
                },
            \"score\":0.694824964
        }
    ]
}

Please note the prjRef value in the metadata.

When I now call the index exactly the same but this time with filtering, I get zero returned vectors in count and or matches:

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

let vectorQuery = await VEC_BAKED_CONTENT_INDEX.query(
            vectors, { 
                topK: 3, 
                filter: {
                    prjRef: contentQueryDataType.projectRef
                }, 
                returnValues: true, 
                returnMetadata: true 
            });

O

Old Man Maker•2/23/24, 8:38 AM

I'm thinking this is because of the way I am creating metedata in the first place?

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

let newMetaData1: MetadataSchemaD1 = {
            d1Key: contentData.d1Key,
            prjRef: contentData.projectRef,
            fltRef: contentData.fleetRef,
            orgRef: contentData.organizationRef,
            scope: contentData.vectorScope,
            vecType: VectorMetaType.CONTENT
        }

        const metaRecord1: Record<string, VectorizeVectorMetadata> = {
            "data": JSON.stringify(newMetaData1)
        };

In my mind, the filtering would be trying to filter on the "data" value without any access to the JSON object held in it?

O

Old Man Maker•2/23/24, 9:13 AM

And yes... my earliest Vectorize code was at fault here. Having pushed the metadata directly into the vector, filtering is working as expected!

DDhravya Shah hey, i've been trying to add article-size embeddings to vectorize, which, predic...

I

Isaac McFadyen•2/28/24, 7:16 PM

When you create the vectorize index you can specify the vector size I believe.

I

Isaac McFadyen•2/28/24, 7:16 PM

There's presets but you can also hit custom and provide your own size.

I

Isaac McFadyen•2/28/24, 7:17 PM

Just note that I believe they expect all vectors to be the same size.

how silly will it be for me to build my website around this vectorize functionality? 😄 i plan on u

Similar Threads

Similar Threads

Similar Threads