Cloudflare Developers•3y ago

Metadata filtering has been announced as a feature launching over the next couple of weeks 🙂

Metadata filtering has been announced as a feature launching over the next couple of weeks

scotto•11/21/23, 1:42 PM

any specific date for metadata filtering? waiting for this to launch

Hhuw Metadata filtering has been announced as a feature launching over the next coupl...

Shravan•11/21/23, 2:41 PM

sweet, do you have a link to the post?

SShravan sweet, do you have a link to the post? 🙂

Matt Silverlock•11/21/23, 2:51 PM

vectorize

MMatt Silverlock https://discord.com/channels/595317990191398933/1152193114522525726/116715426812...

Shravan•11/21/23, 2:59 PM

thanks matt! the meta data feature is a game changer for use cases. Do you know if it is going to be with cross filtering or some other technique like (pre or post filtering)? It would make a huge difference in deciding which vector system to use for different use cases

SShravan thanks matt! the meta data feature is a game changer for use cases. Do you kn...

Matt Silverlock•11/21/23, 3:33 PM

It will be pre filtering.

Frank•11/21/23, 8:42 PM

is namespace filtering working yet?

FFrank is namespace filtering working yet?

huwOP•11/21/23, 10:07 PM

Yes

scotto•11/21/23, 11:10 PM

there is any info the doc reg namespace filtering? can't find it

Sscotto there is any info the doc reg namespace filtering? can't find it

wadefletch•11/22/23, 12:00 AM

vectorize

MMatt Silverlock namespaces are part of the TypeScript/JS API; we haven't documented them yet as ...

scotto•11/22/23, 1:28 AM

bug still there?

Sscotto bug still there?

Matt Silverlock•11/22/23, 1:29 AM

No.

MMatt Silverlock No.

scotto•11/22/23, 1:59 AM

Great, any code example on how to use the namespace filter ?

Sscotto Great, any code example on how to use the namespace filter ?

huwOP•11/22/23, 3:32 AM

Just add the

namespace

namespace

parameter in the body of your

put

put

and

query

query

calls. You can only associate a vector with a single namespace at a time.

Hhuw Just add the `namespace` parameter in the body of your `put` and `query` calls. ...

scotto•11/22/23, 12:50 PM

Great, thanks

Frank•11/23/23, 5:28 PM

whats the char limit of a namespace?

Frank•11/23/23, 5:30 PM

would be cool to bulk delete by namespace

FFrank whats the char limit of a namespace?

Matt Silverlock•11/23/23, 6:01 PM

Matt Silverlock•11/23/23, 6:02 PM

Namespace docs: https://developers.cloudflare.com/vectorize/learning/insert-vectors/#namespaces

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

Sscotto Great, any code example on how to use the namespace filter ?

Matt Silverlock•11/23/23, 6:02 PM

https://developers.cloudflare.com/vectorize/learning/insert-vectors/#namespaces docs!

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

deo time•11/25/23, 6:35 AM

Not really sure why, but it seems like uploading vectors from a file with the CLI ends up losing some entries (should be 68295)

deo time•11/25/23, 6:35 AM

Same here with a 1000 entry file

deo time•11/25/23, 7:33 PM

Also having this happen sometimes when uploading

divby0•11/25/23, 8:43 PM

Trying to make sense of this. Why is it 10'000 PLUS 100? Multiplying the end result by the dimension count makes sense to me but either this should be multiplied (cause you 100 times you run a query against 10k vectors) or something completely differnet but I don't get the plus

divby0•11/25/23, 8:43 PM

Like why is querying those 10k vecotrs once vs ten times not close to ten times the price? Something doesn't add up in my head

Ddivby0 Like why is querying those 10k vecotrs once vs ten times not close to ten times ...

Matt Silverlock•11/25/23, 9:29 PM

It’s just storage (consistent) and then your queries. It is additive. It would be horrendously expensive if it was multiplicative.

divby0•11/25/23, 9:49 PM

yeah if it's storage that makes sense. But it says stored vectors are computed by multiplying stored vectors multiplied by dimensions. And seperately, queried vectors are 10k + 100

divby0•11/25/23, 9:50 PM

just the point "queried vector dimensions" has

your total queried vector dimensions would be 3.878million

your total queried vector dimensions would be 3.878million

which is wrong, isn't it?

Frank•11/25/23, 11:16 PM

is the 1000 namespace limit going to be lifted post beta, or what sort of an increase are we expecting?

deo time•11/26/23, 1:32 AM

Having the same issue

FFrank is the 1000 namespace limit going to be lifted post beta, or what sort of an inc...

huwOP•11/26/23, 5:30 AM

To your first question: Yes, the 1,000 namespace limit is listed as a beta limitation in the Limits page.
To your second question: I am going to heavily speculate here, but I think Vectorize is a layer on top of Durable Objects based on the old million vector demo. It looks like cluster centroids are stored on different Durable Objects, which to me would imply that namespace segmentation is possibly occurring by segmenting each namespace into its own ‘cluster’ of DOs (which would make it faster than metadata filtering, which we have confirmation of above). So the would theoretically be no limit here, or at least the limit would be constrained by how the namespace indexation is occurring.

00xWW Figured out what it was; my requests were exceeding the token limit, as I was us...

huwOP•11/27/23, 1:06 AM

I tried computing token lengths for BAAI but I don’t think we fully support transformers.js yet—would love to know if I’m wrong on that

divby0•11/27/23, 11:37 AM

After Vectorize bindings are not working for Pages Functions, I had to deal with the API of it which is really not documented.

Why are you hiding how the body looks like? Why is the example @/path/to/vectors.ndjson@/path/to/vectors.ndjson not clickable? I have no clue how to continue:

divby0•11/27/23, 11:38 AM

Screenshot is taken from here: https://developers.cloudflare.com/api/operations/vectorize-insert-vector#request-body

I've read all docs of the workers API and the HTTP API now and I think combining the knowledge, I tried sending a POST request with the correct authorization to the insert endpoint.

The body is

{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}
{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}

{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}
{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}

as taken from one of the docs examples. The

Content-Type

Content-Type

headers have been set to application/x-ndjsonapplication/x-ndjson. The server returns "unexpected - Internal server error" with status code 500.

I guess I am still missing something that would be in place when this would be a real file that I was sending in the post body.

Update: I have tested adding it as a file in Postman just to try it out. Still not working :/

Update 2: The 500 internal error was due to me testing with 3 dimensional vectors but my actual index had way more. Solution: Test with the vectors of the same dimension of course...

scotto•11/27/23, 12:31 PM

hi, namespace filtering is not working for me, have a short code on how to reproduce

Indexing and searching, if i remove the namespace param from the search file, results are there, but using namespace filtering i'm not getting any results (with the same vector)

indexing.js33.04KB

Ccosbgn That would be amazing, it's quite complicated now to deal with situation where a...

divby0•11/27/23, 12:54 PM

This. It's so important to be able to delete again, as there is the 1000 namespace limit (each of my users creates several namespaces but usually deletes the related objects in the DB again).

Like the 1000 namespace thing wouldn't be such an issue if we could delete some again.

divby0•11/27/23, 1:24 PM

Idea: It would be nice to be able to query vectors and get the metadata but NOT the vectors (since my metadata is way shorter than the 1536 dimensional vector)

Ddivby0 Idea: It would be nice to be able to query vectors and get the metadata but NOT...

zegevlier•11/27/23, 1:27 PM

This is now possible with the returnMetadata: truereturnMetadata: true parameter, as listed in https://developers.cloudflare.com/vectorize/platform/client-api/#query-vectors

divby0•11/27/23, 1:28 PM

Ah awesome! That definitely needs to be added to https://developers.cloudflare.com/api/operations/vectorize-query-vector doc for HTTP

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

divby0•11/27/23, 2:22 PM

I have 1536 dimension openai embeddings and I can't insert 284 of them, but it works if I only insert the first up to 100 vectors. 101 doesn't work.

I looked up the limitations and didn't see anything relevant. Is this a bug?

Update: I implemented chunking and now all the request fail (they have 90 vectors each, so I have like 4 of them). If I try a few times, sometimes several of the 4 request succeed, but never all and it's not reliably of course

Update 2: Still trying to triangulate the issue. Tested by slicing only the first 3 dimensions and then I can insert all at once without a problem or batch them into 4 batches without a problem. Weird

Update 3: Batching into maxSize 10 batches (29 of them) did work on all 29 insert requests. Is there any metadata or index name limit per batch maybe?

Ddivby0 I have 1536 dimension openai embeddings and I can't insert 284 of them, but it w...

Matt Silverlock•11/27/23, 3:12 PM

Do you have metadata or index names that exceed the limits? Are you positive they don’t?

MMatt Silverlock Do you have metadata or index names that exceed the limits? Are you positive the...

divby0•11/27/23, 3:13 PM

Otherwise batching them into 29 individual batches wouldn't have 100% success rate

divby0•11/27/23, 3:13 PM

Let me programmatically check again

divby0•11/27/23, 3:15 PM

id is a uuid-v4 (36 characters)
and the metadata is always below 8000 characters for all 284 vectors

Ddivby0 id is a uuid-v4 (36 characters) and the metadata is always below 8000 characters...

Matt Silverlock•11/27/23, 4:15 PM

How are you counting metadata length? The limit is enforced on all bytes (keys, etc included), not just values. Our error handling needs to be better here but would be good to rule that out.

Ddivby0 Otherwise batching them into 29 individual batches wouldn't have 100% success ra...

deo time•11/27/23, 4:31 PM

I had to write a helper that would batch them into groups of 100 as well as retry if a 500 error happened, worked for the most part

deo time•11/27/23, 4:34 PM

If you're inserting a lot though itll take quite a while

Metadata filtering has been announced as a feature launching over the next couple of weeks 🙂

Similar Threads