Cloudflare Developers•3y ago

Just to confirm, is this the right way to bind many indexes to a worker?

JJuanWTF Just to confirm, is this the right way to bind many indexes to a worker?

huw•11/16/23, 10:51 PM

Yep!

huw•11/16/23, 10:52 PM

Multiple indices vs multiple namespaces will depend on what you’re trying to achieve

JuanWTFOP•11/16/23, 10:59 PM

What would be multiple namespace in this context?

MMatt Silverlock Can you share your index ID here?

Tumbzilla•11/17/23, 2:10 PM

hey @Matt - Any luck identifying the cause of the issue? Is the problem something I'm doing on my side?

Jerome•11/17/23, 9:39 PM

@Tumbzilla DM'd you

JJuanWTF What would be multiple namespace in this context?

huw•11/18/23, 1:38 AM

Using the

namespace

namespace

option when inserting & querying vectors (undocumented beta feature)

huw•11/18/23, 1:38 AM

It’s just a simple way to segment your index

GuillaumeLakano•11/18/23, 9:20 AM

Hi! How is priced getByIds() calls please? It's not really a querying, and from what I understood the Pricing is based on stored content and querying, but not about the operation that insert inside Vectorise.

GGuillaumeLakano Hi! How is priced getByIds() calls please? It's not really a querying, and from ...

Matt Silverlock•11/18/23, 4:13 PM

It is still a query. You are just scoping it to specific IDs.

scotto•11/18/23, 4:31 PM

metadata filtering will be announced here? any ETA?

steyblind•11/19/23, 12:23 AM

what bit do I need to flip to pick up the latest types?

steyblind•11/19/23, 12:23 AM

compatibility_date = "2023-11-08"

compatibility_date = "2023-11-08"

kian•11/19/23, 12:47 AM

There hasn’t been a workerd/types release since that functionality was added

kian•11/19/23, 12:57 AM

At a glance, I’m also not sure if the flag is actually available yet.

kian•11/19/23, 12:57 AM

I’m not at a device to test, but the last attempt I can see shows

No such compatibility flag: vectorize_query_metadata_optional

No such compatibility flag: vectorize_query_metadata_optional

steyblind•11/19/23, 2:24 AM

Gotcha. Saw it here https://developers.cloudflare.com/vectorize/learning/query-vectors/

Query vectors · Vectorize

Querying an index, or vector search, enables you to search an index by providing an input vector and returning the nearest vectors based on the …

GuillaumeLakano•11/19/23, 2:28 PM

Hi, for our project, we need to known how we can structure our vectors, as currently in beta it's limited to 200,000 vectors / index.
Technically we only need one index, but ideally unlimited vectors.
Could you tell us if we can expect at least thousand of millions vectors / index in GA, or if we need right now to split our vectors inside multiple indexes, and queries each individual indexes to have our final results ?

GGuillaumeLakano Hi, for our project, we need to known how we can structure our vectors, as curre...

Matt Silverlock•11/19/23, 8:50 PM

vectorize

GGuillaumeLakano Hi, for our project, we need to known how we can structure our vectors, as curre...

Matt Silverlock•11/19/23, 8:51 PM

If you need “thousands of millions” (billions) per index, then you should absolutely split across multiple indexes. Running a 1B+ vector index on Pinecone or elsewhere would also be $$$$

Shravan•11/21/23, 11:53 AM

Will there be a way to cross filter by vector and meta? Currently this is not supported

SShravan Will there be a way to cross filter by vector and meta? Currently this is not s...

huw•11/21/23, 12:47 PM

Metadata filtering has been announced as a feature launching over the next couple of weeks

scotto•11/21/23, 1:42 PM

any specific date for metadata filtering? waiting for this to launch

Hhuw Metadata filtering has been announced as a feature launching over the next coupl...

Shravan•11/21/23, 2:41 PM

sweet, do you have a link to the post?

SShravan sweet, do you have a link to the post? 🙂

Matt Silverlock•11/21/23, 2:51 PM

vectorize

MMatt Silverlock https://discord.com/channels/595317990191398933/1152193114522525726/116715426812...

Shravan•11/21/23, 2:59 PM

thanks matt! the meta data feature is a game changer for use cases. Do you know if it is going to be with cross filtering or some other technique like (pre or post filtering)? It would make a huge difference in deciding which vector system to use for different use cases

SShravan thanks matt! the meta data feature is a game changer for use cases. Do you kn...

Matt Silverlock•11/21/23, 3:33 PM

It will be pre filtering.

Frank•11/21/23, 8:42 PM

is namespace filtering working yet?

FFrank is namespace filtering working yet?

huw•11/21/23, 10:07 PM

Yes

scotto•11/21/23, 11:10 PM

there is any info the doc reg namespace filtering? can't find it

Sscotto there is any info the doc reg namespace filtering? can't find it

wadefletch•11/22/23, 12:00 AM

vectorize

MMatt Silverlock namespaces are part of the TypeScript/JS API; we haven't documented them yet as ...

scotto•11/22/23, 1:28 AM

bug still there?

Sscotto bug still there?

Matt Silverlock•11/22/23, 1:29 AM

No.

MMatt Silverlock No.

scotto•11/22/23, 1:59 AM

Great, any code example on how to use the namespace filter ?

Sscotto Great, any code example on how to use the namespace filter ?

huw•11/22/23, 3:32 AM

Just add the

namespace

namespace

parameter in the body of your

put

put

and

query

query

calls. You can only associate a vector with a single namespace at a time.

Hhuw Just add the `namespace` parameter in the body of your `put` and `query` calls. ...

scotto•11/22/23, 12:50 PM

Great, thanks

Frank•11/23/23, 5:28 PM

whats the char limit of a namespace?

Frank•11/23/23, 5:30 PM

would be cool to bulk delete by namespace

FFrank whats the char limit of a namespace?

Matt Silverlock•11/23/23, 6:01 PM

Matt Silverlock•11/23/23, 6:02 PM

Namespace docs: https://developers.cloudflare.com/vectorize/learning/insert-vectors/#namespaces

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

Sscotto Great, any code example on how to use the namespace filter ?

Matt Silverlock•11/23/23, 6:02 PM

https://developers.cloudflare.com/vectorize/learning/insert-vectors/#namespaces docs!

Insert vectors · Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains …

deo time•11/25/23, 6:35 AM

Not really sure why, but it seems like uploading vectors from a file with the CLI ends up losing some entries (should be 68295)

deo time•11/25/23, 6:35 AM

Same here with a 1000 entry file

deo time•11/25/23, 7:33 PM

Also having this happen sometimes when uploading

divby0•11/25/23, 8:43 PM

Trying to make sense of this. Why is it 10'000 PLUS 100? Multiplying the end result by the dimension count makes sense to me but either this should be multiplied (cause you 100 times you run a query against 10k vectors) or something completely differnet but I don't get the plus

divby0•11/25/23, 8:43 PM

Like why is querying those 10k vecotrs once vs ten times not close to ten times the price? Something doesn't add up in my head

Ddivby0 Like why is querying those 10k vecotrs once vs ten times not close to ten times ...

Matt Silverlock•11/25/23, 9:29 PM

It’s just storage (consistent) and then your queries. It is additive. It would be horrendously expensive if it was multiplicative.

divby0•11/25/23, 9:49 PM

yeah if it's storage that makes sense. But it says stored vectors are computed by multiplying stored vectors multiplied by dimensions. And seperately, queried vectors are 10k + 100

divby0•11/25/23, 9:50 PM

just the point "queried vector dimensions" has

your total queried vector dimensions would be 3.878million

your total queried vector dimensions would be 3.878million

which is wrong, isn't it?

Frank•11/25/23, 11:16 PM

is the 1000 namespace limit going to be lifted post beta, or what sort of an increase are we expecting?

Just to confirm, is this the right way to bind many indexes to a worker?

Similar Threads

Similar Threads

Similar Threads