Hi! How is priced getByIds() calls please? It's not really a querying, and from what I understood th

Hi! How is priced getByIds() calls please? It's not really a querying, and from what I understood the Pricing is based on stored content and querying, but not about the operation that insert inside Vectorise.

GGuillaumeLakano Hi! How is priced getByIds() calls please? It's not really a querying, and from ...

Matt Silverlock•11/18/23, 4:13 PM

It is still a query. You are just scoping it to specific IDs.

scotto•11/18/23, 4:31 PM

metadata filtering will be announced here? any ETA?

steyblind•11/19/23, 12:23 AM

what bit do I need to flip to pick up the latest types?

steyblind•11/19/23, 12:23 AM

compatibility_date = "2023-11-08"

compatibility_date = "2023-11-08"

kian•11/19/23, 12:47 AM

There hasn’t been a workerd/types release since that functionality was added

kian•11/19/23, 12:57 AM

At a glance, I’m also not sure if the flag is actually available yet.

kian•11/19/23, 12:57 AM

I’m not at a device to test, but the last attempt I can see shows

No such compatibility flag: vectorize_query_metadata_optional

No such compatibility flag: vectorize_query_metadata_optional

steyblind•11/19/23, 2:24 AM

Gotcha. Saw it here https://developers.cloudflare.com/vectorize/learning/query-vectors/

Query vectors · Vectorize

Querying an index, or vector search, enables you to search an index by providing an input vector and returning the nearest vectors based on the …

GuillaumeLakanoOP•11/19/23, 2:28 PM

Hi, for our project, we need to known how we can structure our vectors, as currently in beta it's limited to 200,000 vectors / index.
Technically we only need one index, but ideally unlimited vectors.
Could you tell us if we can expect at least thousand of millions vectors / index in GA, or if we need right now to split our vectors inside multiple indexes, and queries each individual indexes to have our final results ?

GGuillaumeLakano Hi, for our project, we need to known how we can structure our vectors, as curre...

Matt Silverlock•11/19/23, 8:50 PM

vectorize

GGuillaumeLakano Hi, for our project, we need to known how we can structure our vectors, as curre...

Matt Silverlock•11/19/23, 8:51 PM

If you need “thousands of millions” (billions) per index, then you should absolutely split across multiple indexes. Running a 1B+ vector index on Pinecone or elsewhere would also be $$$$

Shravan•11/21/23, 11:53 AM

Will there be a way to cross filter by vector and meta? Currently this is not supported

SShravan Will there be a way to cross filter by vector and meta? Currently this is not s...

huw•11/21/23, 12:47 PM

Metadata filtering has been announced as a feature launching over the next couple of weeks

scotto•11/21/23, 1:42 PM

any specific date for metadata filtering? waiting for this to launch

Hhuw Metadata filtering has been announced as a feature launching over the next coupl...

Shravan•11/21/23, 2:41 PM

sweet, do you have a link to the post?

SShravan sweet, do you have a link to the post? 🙂

Matt Silverlock•11/21/23, 2:51 PM

vectorize

MMatt Silverlock https://discord.com/channels/595317990191398933/1152193114522525726/116715426812...

Shravan•11/21/23, 2:59 PM

thanks matt! the meta data feature is a game changer for use cases. Do you know if it is going to be with cross filtering or some other technique like (pre or post filtering)? It would make a huge difference in deciding which vector system to use for different use cases

SShravan thanks matt! the meta data feature is a game changer for use cases. Do you kn...

Matt Silverlock•11/21/23, 3:33 PM

It will be pre filtering.

Frank•11/21/23, 8:42 PM

is namespace filtering working yet?

FFrank is namespace filtering working yet?

huw•11/21/23, 10:07 PM

Yes

scotto•11/21/23, 11:10 PM

there is any info the doc reg namespace filtering? can't find it

Sscotto there is any info the doc reg namespace filtering? can't find it

wadefletch•11/22/23, 12:00 AM

vectorize

MMatt Silverlock namespaces are part of the TypeScript/JS API; we haven't documented them yet as ...

scotto•11/22/23, 1:28 AM

bug still there?

Sscotto bug still there?

Matt Silverlock•11/22/23, 1:29 AM

No.

MMatt Silverlock No.

scotto•11/22/23, 1:59 AM

Great, any code example on how to use the namespace filter ?

Sscotto Great, any code example on how to use the namespace filter ?

huw•11/22/23, 3:32 AM

Just add the

namespace

namespace

parameter in the body of your putput and

query

query

calls. You can only associate a vector with a single namespace at a time.

Hhuw Just add the `namespace` parameter in the body of your `put` and `query` calls. ...

scotto•11/22/23, 12:50 PM

Great, thanks

Frank•11/23/23, 5:28 PM

whats the char limit of a namespace?

Frank•11/23/23, 5:30 PM

would be cool to bulk delete by namespace

FFrank whats the char limit of a namespace?

Matt Silverlock•11/23/23, 6:01 PM

Matt Silverlock•11/23/23, 6:02 PM

Namespace docs: https://developers.cloudflare.com/vectorize/learning/insert-vectors/#namespaces

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

Sscotto Great, any code example on how to use the namespace filter ?

Matt Silverlock•11/23/23, 6:02 PM

https://developers.cloudflare.com/vectorize/learning/insert-vectors/#namespaces docs!

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

deo time•11/25/23, 6:35 AM

Not really sure why, but it seems like uploading vectors from a file with the CLI ends up losing some entries (should be 68295)

deo time•11/25/23, 6:35 AM

Same here with a 1000 entry file

deo time•11/25/23, 7:33 PM

Also having this happen sometimes when uploading

divby0•11/25/23, 8:43 PM

Trying to make sense of this. Why is it 10'000 PLUS 100? Multiplying the end result by the dimension count makes sense to me but either this should be multiplied (cause you 100 times you run a query against 10k vectors) or something completely differnet but I don't get the plus

divby0•11/25/23, 8:43 PM

Like why is querying those 10k vecotrs once vs ten times not close to ten times the price? Something doesn't add up in my head

Ddivby0 Like why is querying those 10k vecotrs once vs ten times not close to ten times ...

Matt Silverlock•11/25/23, 9:29 PM

It’s just storage (consistent) and then your queries. It is additive. It would be horrendously expensive if it was multiplicative.

divby0•11/25/23, 9:49 PM

yeah if it's storage that makes sense. But it says stored vectors are computed by multiplying stored vectors multiplied by dimensions. And seperately, queried vectors are 10k + 100

divby0•11/25/23, 9:50 PM

just the point "queried vector dimensions" has your total queried vector dimensions would be 3.878millionyour total queried vector dimensions would be 3.878million which is wrong, isn't it?

Frank•11/25/23, 11:16 PM

is the 1000 namespace limit going to be lifted post beta, or what sort of an increase are we expecting?

deo time•11/26/23, 1:32 AM

Having the same issue

FFrank is the 1000 namespace limit going to be lifted post beta, or what sort of an inc...

huw•11/26/23, 5:30 AM

To your first question: Yes, the 1,000 namespace limit is listed as a beta limitation in the Limits page.
To your second question: I am going to heavily speculate here, but I think Vectorize is a layer on top of Durable Objects based on the old million vector demo. It looks like cluster centroids are stored on different Durable Objects, which to me would imply that namespace segmentation is possibly occurring by segmenting each namespace into its own ‘cluster’ of DOs (which would make it faster than metadata filtering, which we have confirmation of above). So the would theoretically be no limit here, or at least the limit would be constrained by how the namespace indexation is occurring.

00xWW Figured out what it was; my requests were exceeding the token limit, as I was us...

huw•11/27/23, 1:06 AM

I tried computing token lengths for BAAI but I don’t think we fully support transformers.js yet—would love to know if I’m wrong on that

divby0•11/27/23, 11:37 AM

After Vectorize bindings are not working for Pages Functions, I had to deal with the API of it which is really not documented.

Why are you hiding how the body looks like? Why is the example @/path/to/vectors.ndjson@/path/to/vectors.ndjson not clickable? I have no clue how to continue:

divby0•11/27/23, 11:38 AM

Screenshot is taken from here: https://developers.cloudflare.com/api/operations/vectorize-insert-vector#request-body

I've read all docs of the workers API and the HTTP API now and I think combining the knowledge, I tried sending a POST request with the correct authorization to the insert endpoint.

The body is

{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}
{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}

{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}
{"id":"1","values":[32.4,74.1,3.2],"metadata":{"path":"r2://bucket-name/path/to/image.png","format":"png","category":"profile_image"}}

as taken from one of the docs examples. The

Content-Type

Content-Type

headers have been set to application/x-ndjsonapplication/x-ndjson. The server returns "unexpected - Internal server error" with status code 500.

I guess I am still missing something that would be in place when this would be a real file that I was sending in the post body.

Update: I have tested adding it as a file in Postman just to try it out. Still not working :/

Update 2: The 500 internal error was due to me testing with 3 dimensional vectors but my actual index had way more. Solution: Test with the vectors of the same dimension of course...

Hi! How is priced getByIds() calls please? It's not really a querying, and from what I understood th

Similar Threads

Similar Threads

Similar Threads