It's possible to create metadata filters after index creation, though metadata search will not apply

It's possible to create metadata filters after index creation, though metadata search will not apply to already existing data.

Kkingmesal I'm very eager to move off of pinecone, purely from a simplicity perspective. ...

yevgenOP•10/14/24, 10:26 AM

re #2

an actual view in the dashboard would be a big help

an actual view in the dashboard would be a big help

Index analytics is available in dash

HHexMan Is there any way to add more than 1 namespace to a vector that I am inserting? ...

yevgenOP•10/14/24, 10:31 AM

You could use https://developers.cloudflare.com/vectorize/best-practices/insert-vectors/#metadata to add multiple fields describing vectors and index each of the metadata fields separately to support filtering.

Cloudflare Docs

Insert vectors | Vectorize

Vectorize indexes allow you to insert vectors at any point: Vectorize will optimize the index behind the scenes to ensure that vector search remains efficient, even as new vectors are added or existing vectors updated.

Kkingmesal I'm very eager to move off of pinecone, purely from a simplicity perspective. ...

yevgenOP•10/14/24, 10:33 AM

Metadata filtering has to support <, <=, >=, > ... Even a filter for in list or not in list.

Could you please list the order of metadata filtering operations you need the most to get started?

IIsaac McFadyen And nothing has *zero* limits: Pinecone docs say they have them too, it just sca...

yevgenOP•10/14/24, 10:42 AM

That is so true regarding nothing has zero limits. We are happy to work with customers to increase most of the limits once they are close to hit them.

Yyevgen re #2 `an actual view in the dashboard would be a big help` Index analytics is a...

kingmesal•10/14/24, 1:38 PM

Fair point. Those have been there. I meant a way to view / query something from the dashboard, similar to KV, or D1. An abiltity to perform a sanity check. e.g. In pinecone I had a bug once where I was removing a bunch of vectors before inserting a batch up new records and I had a bug, it died before inserting the new records. In their dashboard I had the ability to see some basic in formation to know that the data was gone.

Yyevgen > Metadata filtering has to support <, <=, >=, > ... Even a filter for in list o...

kingmesal•10/14/24, 1:46 PM

Sure, here are a couple of the examples, in priority order.

I have a few different use cases for putting dates (number format, seconds of ms is normal) on the vectors as a way to only bring back candidates after (>=) X and before (<) Y. Basically, Temporal limitations.
I have a list of arbitrary attributes, call it categorical information. Filter vectors where X is in that list. Or filter vectors where X is not in that list.

Jerome•10/17/24, 9:13 AM

Hi there! Now that V2 has GA'd, we will disable the creation of V1 indexes during this quarter. When this change will be effective, existing V1 indexes will be unaffected and continue operating as normal.

JJerome Hi there! Now that V2 has GA'd, we will disable the **creation** of V1 indexes d...

Isaac McFadyen•10/17/24, 2:41 PM

Great to hear! Once it happens (or perhaps a bit ahead) can it be added to https://developers.cloudflare.com/vectorize/platform/changelog/ as well?

Keebs•10/18/24, 12:08 PM

anyone try to create the embedding to query mongodb vendor search?

Yyevgen It's possible to create metadata filters after index creation, though metadata s...

EndiM•10/19/24, 6:26 AM

Hi. Would it apply if i were to upsert on that existing data? So if i create a new metadata index, and upsert the data, can that metadata be used as a filter? Since it overrides, im guessing yes, but i need to make sure!

EndiM•10/19/24, 6:30 AM

Also, one other question. On the filter operation there is only eq and ne. Is there a way to use contains logic? Ex: you search for an address, but on the results its not in the 20 topk. If i were to have a containy on the adress metadata tho, i would force my vector search to isolate my responses on this search option.

Daniel•10/19/24, 5:38 PM

Hi all! I have a beginner question. I am trying to use Vectorize with my NextJS Cloudflare Pages website. In this tutorial https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/ they say to use

npx wrangler dev --remote

npx wrangler dev --remote

to run locally and iterate.

When I run that, I see:

✘ [ERROR] It looks like you've run a Workers-specific command in a Pages project.

  For Pages, please run `wrangler pages dev` instead.

✘ [ERROR] It looks like you've run a Workers-specific command in a Pages project.

  For Pages, please run `wrangler pages dev` instead.

But when I do that, I see:

❯ npx wrangler pages dev --remote


✘ [ERROR] Unknown argument: remote


wrangler pages dev [directory] [-- command..]

Develop your full-stack Pages application locally

❯ npx wrangler pages dev --remote


✘ [ERROR] Unknown argument: remote


wrangler pages dev [directory] [-- command..]

Develop your full-stack Pages application locally

I need to use

--remote

--remote

because of:

▲ [WARNING] Vectorize bindings are not currently supported in local mode. Please use --remote if you are working with them.

▲ [WARNING] Vectorize bindings are not currently supported in local mode. Please use --remote if you are working with them.

And without that, I see:

✘ [ERROR] Error in GET function: Error: VECTOR_INDEX binding is not available

✘ [ERROR] Error in GET function: Error: VECTOR_INDEX binding is not available

The AI binding and KV binding are working fine, no issues there, but I can't seem to get the Vectorize binding to show up at all. Maybe I'm making an obvious mistake -- I'm somewhat new to Cloudflare. Any help appreciated, thank you!

EEndiM Hi. Would it apply if i were to upsert on that existing data? So if i create a n...

yevgenOP•10/21/24, 1:31 PM

yes, it will apply, we updated dev docs to reflect that https://developers.cloudflare.com/vectorize/reference/metadata-filtering/#metadata-indexes

Cloudflare Docs

Metadata filtering | Vectorize

In addition to providing an input vector to your query, you can also filter by vector metadata associated with every vector. Query results will only include vectors that match the filter criteria, meaning that filter is applied first, and the topK results are taken from the filtered set.

EEndiM Also, one other question. On the filter operation there is only eq and ne. Is th...

yevgenOP•10/21/24, 1:36 PM

there is no support for contains. We are planning add support for the following operations:

greater than
- greater than or equal
- less than
- less than or equal
- in s specified array
- not in a specified array

Rahul gangotri•10/21/24, 3:27 PM

is this approach is good of create a unique NS for each user i have and keep their vectors in that namespace?
and multiple indexes for set of users

why is it onnly limited to 100 indexes per account, can we request to increase this?

Rahul gangotri•10/21/24, 3:27 PM

the product i am planning is going to store a lot of vectors

Rahul gangotri•10/21/24, 3:28 PM

we are already managing huge number of vectors using qdrant self hosted, but looking for cloudflare solution

Rahul gangotri•10/21/24, 3:32 PM

anyone?

RRahul gangotri is this approach is good of create a unique NS for each user i have and keep the...

yevgenOP•10/22/24, 12:01 PM

We are increasing limits (# of indexes per account and # of namespaces per index) for Workers Paid plan this week, stay tuned.

Rahul gangotri•10/22/24, 12:02 PM

RRahul gangotri is this approach is good of create a unique NS for each user i have and keep the...

Chinoman10•10/22/24, 3:15 PM

If users don't need to access each other's vectors (if they should be silo'ed), then yes, creating a namespace per user is OK, although right-now there's a 1k namespaces limit per index.
If they should be able to share them, then they should share a namespace and you use metadata filters instead for user segregation.

So at the end of the day, you'll have to think what suits your use-case the best.

Yyevgen there is no support for contains. We are planning add support for the following ...

Chinoman10•10/22/24, 3:30 PM

Also, any ETA for this?

CChinoman10 Also, any ETA for this?

yevgenOP•10/23/24, 8:48 AM

we will be aiming for mid Q1

yevgenOP•10/24/24, 2:34 PM

We have increased Vectorize limits for v2 as per https://developers.cloudflare.com/vectorize/platform/limits/

Cloudflare Docs

Limits | Vectorize

The following limits apply to accounts, indexes and vectors (as specified):

yanlin•10/25/24, 6:49 PM

For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ?

another question is that any upsert support that only updates the metadata based on id ? Whats the recommended way to update metadata?

freddw•10/26/24, 7:27 PM

Is best practice for supporting more complicated query semantics than just filtering to upsert vectors with a metadata field containing foreign keys to D1 records? Then implement your own pre/post filtering between the two?

freddw•10/26/24, 7:32 PM

as an aside it seems like if the reasoning behind launching vectorize as a separate product from D1 is that either sqlite did not support vector search at the time of launch, or that it was not considered an ideal substrate for vector search, it could be nice to integrate vectorize within D1 under the hood (with restrictions to what is possible within vectorize databases) so that users could write simpler queries/have easier setup/less to manage. Because at the end of the day a lot of people (like me) will want to combine topK with joins and it'd be nice to not have to implement some of that myself, assuming I'm not missing something re: the current impl

yanlin•10/28/24, 12:22 AM

Given the limitations of the meta-filtering operation, I have to leverage a nested key structure as documented:
{ "pandas.nice": 42 }
// looks for { "pandas": { "nice": 42 } }

However, when I attempt to upsert metadata where the key contains a JSON object, as specified in the documentation, I encounter the following error:

VECTOR_UPSERT_ERROR (code = 40019): invalid metadata for vector id=“jhbqpwft0qogbr1fs6re-0”; the metadata JSON object cannot contain JSON objects.

I’m not sure what to do next. Does anyone have any suggestions? Thanks!

JJerome Hi there! Now that V2 has GA'd, we will disable the **creation** of V1 indexes d...

Talljoe•10/28/24, 6:56 AM

When do you expect to update the Typescript API to use V2? I'm running into a problem where Wrangler doesn't support the

--deprecated-v1

--deprecated-v1

flag on certain calls. I just created the index today so I'm very happy to use V2 but don't want to roll my own client.

npm

cloudflare

The official TypeScript library for the Cloudflare API. Latest version: 3.5.0, last published: 3 months ago. Start using cloudflare in your project by running

npm i cloudflare

npm i cloudflare

. There are 93 other projects in the npm registry using cloudflare.

Yyanlin For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? anoth...

yevgenOP•10/28/24, 11:28 AM

For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ?

Yes

Yyanlin For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? anoth...

yevgenOP•10/28/24, 11:29 AM

another question is that any upsert support that only updates the metadata based on id ? Whats the recommended way to update metadata?

It's on a roadmap to provide

Jerome•10/28/24, 12:15 PM

We posted a deep dive into how we are building Vectorize on Cloudflare's blog. https://blog.cloudflare.com/building-vectorize-a-distributed-vector-database-on-cloudflare-developer-platform/ Check it out

The Cloudflare Blog

Building Vectorize, a distributed vector database, on Cloudflare’s ...

Vectorize was recently upgraded and made generally available, now supporting indexes of up to 5 million vectors, delivering faster responses, with lower pricing and a free tier. This post dives deep into how we built Vectorize to enable these improvements.

Yyevgen > For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? Yes

yanlin•10/28/24, 7:26 PM

thanks. Seems not null is not functional, for example, for any not indexed properties , if i use {"$ne","null"}, it always return everything

Yyanlin thanks. Seems not null is not functional, for example, for any not indexed prope...

Isaac McFadyen•10/28/24, 9:21 PM

Are you quoting

null

null

like that?

Isaac McFadyen•10/28/24, 9:21 PM

Because that's a string

"null"

"null"

, not actual null.

Isaac McFadyen•10/28/24, 9:22 PM

Whereas your initial question wasn't quoted, meaning it would be actual

null

null

IIsaac McFadyen Whereas your initial question wasn't quoted, meaning it would be actual `null`

yanlin•10/28/24, 11:24 PM

sorry for the typo i did {"path": {"$ne": null}, } and the path prop is not indexed

Rahul gangotri•10/29/24, 7:13 AM

Quick question

Why do we need to create metadata index and even if we have to create, why only 1 key at a time and upto 10 index is the limit? While upstash allows to filter on metadata normally on all metadata
Can we also expect glob and more type of searches operators along with eq and neq

Like : contains , IN, not in, glob, pattern match, and , or etc

can we also have hybrid search like a cosine similarity search with Full text search as well for better search in future?

leesus•10/29/24, 10:15 AM

Any updates/timeline on the automatic migration tool from v1 to v2 databases?

yevgenOP•10/30/24, 11:20 AM

Vectorize now has a significantly improved query latency through REST API https://developers.cloudflare.com/vectorize/platform/changelog/#2024-10-28

Cloudflare Docs

Changelog | Vectorize

EndiM•10/30/24, 9:07 PM

Hey! Whats the limit on insert/upsert.
for some reasons, my vector inserts/upsers have stoped at 128.04k vectors.
In dev, in another vectorize database, everything is working fine. Is there any hidden limit we dont know about???

EndiM•10/30/24, 9:10 PM

@yevgen

Yota•10/30/24, 9:25 PM

Is there a way to dynamically add Vectorize bindings programatically?

YYota Is there a way to dynamically add Vectorize bindings programatically?

Yota•10/30/24, 9:27 PM

ah i found it https://developers.cloudflare.com/vectorize/best-practices/create-indexes/

Yota•10/30/24, 9:30 PM

Hmmm, but these don't seem to be enough number for my use-case
Indexes per account 50,000 (Workers Paid)

EndiM•10/31/24, 12:26 AM

This issue is happening from : 2024-10-25T14:36:55.873Z
whatever improvement was made on this date, caused this to break.
Triggering it from binding or api, even tho the call return succees true, it actually does nothing on the index. No delete, no insert, no upset

EEndiM Hey! Whats the limit on insert/upsert. for some reasons, my vector inserts/upse...

yevgenOP•10/31/24, 8:52 AM

Nothing should have caused such issues, let me check with the team, it could be delayed as most of the team is in US time zone.

Yyevgen Nothing should have caused such issues, let me check with the team, it could be ...

EndiM•10/31/24, 8:52 AM

Thanks for getting back! Okay thank you.

yevgenOP•11/1/24, 1:16 PM

Vectorize update:

2024-10-30: Vectorize has local mode support in Wrangler @3.84.0 https://github.com/cloudflare/workers-sdk/releases/tag/wrangler%403.84.0

GitHub

Release wrangler@3.84.0 · cloudflare/workers-sdk

Minor Changes

#6999 0111edb Thanks @garvit-gupta! - docs: Vectorize GA Announcement Banner

#6916 a33a133 Thanks @garrettgu10! - Local development now supports Vectorize bindings

#7004 15ef013...

It's possible to create metadata filters after index creation, though metadata search will not apply

Similar Threads

Similar Threads

Similar Threads