Also, one other question. On the filter operation there is only eq and ne. Is there a way to use con

Also, one other question. On the filter operation there is only eq and ne. Is there a way to use contains logic? Ex: you search for an address, but on the results its not in the 20 topk. If i were to have a containy on the adress metadata tho, i would force my vector search to isolate my responses on this search option.

Daniel•10/19/24, 5:38 PM

Hi all! I have a beginner question. I am trying to use Vectorize with my NextJS Cloudflare Pages website. In this tutorial https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/ they say to use npx wrangler dev --remotenpx wrangler dev --remote to run locally and iterate.

When I run that, I see:

✘ [ERROR] It looks like you've run a Workers-specific command in a Pages project.

  For Pages, please run `wrangler pages dev` instead.

✘ [ERROR] It looks like you've run a Workers-specific command in a Pages project.

  For Pages, please run `wrangler pages dev` instead.

But when I do that, I see:

❯ npx wrangler pages dev --remote


✘ [ERROR] Unknown argument: remote


wrangler pages dev [directory] [-- command..]

Develop your full-stack Pages application locally

❯ npx wrangler pages dev --remote


✘ [ERROR] Unknown argument: remote


wrangler pages dev [directory] [-- command..]

Develop your full-stack Pages application locally

I need to use

--remote

--remote

because of:

▲ [WARNING] Vectorize bindings are not currently supported in local mode. Please use --remote if you are working with them.

▲ [WARNING] Vectorize bindings are not currently supported in local mode. Please use --remote if you are working with them.

And without that, I see:

✘ [ERROR] Error in GET function: Error: VECTOR_INDEX binding is not available

✘ [ERROR] Error in GET function: Error: VECTOR_INDEX binding is not available

The AI binding and KV binding are working fine, no issues there, but I can't seem to get the Vectorize binding to show up at all. Maybe I'm making an obvious mistake -- I'm somewhat new to Cloudflare. Any help appreciated, thank you!

EEndiM Hi. Would it apply if i were to upsert on that existing data? So if i create a n...

yevgen•10/21/24, 1:31 PM

yes, it will apply, we updated dev docs to reflect that https://developers.cloudflare.com/vectorize/reference/metadata-filtering/#metadata-indexes

Cloudflare Docs

Metadata filtering | Vectorize

In addition to providing an input vector to your query, you can also filter by vector metadata associated with every vector. Query results will only include vectors that match the filter criteria, meaning that filter is applied first, and the topK results are taken from the filtered set.

EEndiM Also, one other question. On the filter operation there is only eq and ne. Is th...

yevgen•10/21/24, 1:36 PM

there is no support for contains. We are planning add support for the following operations:

greater than
- greater than or equal
- less than
- less than or equal
- in s specified array
- not in a specified array

Rahul gangotri•10/21/24, 3:27 PM

is this approach is good of create a unique NS for each user i have and keep their vectors in that namespace?
and multiple indexes for set of users

why is it onnly limited to 100 indexes per account, can we request to increase this?

Rahul gangotri•10/21/24, 3:27 PM

the product i am planning is going to store a lot of vectors

Rahul gangotri•10/21/24, 3:28 PM

we are already managing huge number of vectors using qdrant self hosted, but looking for cloudflare solution

Rahul gangotri•10/21/24, 3:32 PM

anyone?

RRahul gangotri is this approach is good of create a unique NS for each user i have and keep the...

yevgen•10/22/24, 12:01 PM

We are increasing limits (# of indexes per account and # of namespaces per index) for Workers Paid plan this week, stay tuned.

Rahul gangotri•10/22/24, 12:02 PM

RRahul gangotri is this approach is good of create a unique NS for each user i have and keep the...

Chinoman10•10/22/24, 3:15 PM

If users don't need to access each other's vectors (if they should be silo'ed), then yes, creating a namespace per user is OK, although right-now there's a 1k namespaces limit per index.
If they should be able to share them, then they should share a namespace and you use metadata filters instead for user segregation.

So at the end of the day, you'll have to think what suits your use-case the best.

Yyevgen there is no support for contains. We are planning add support for the following ...

Chinoman10•10/22/24, 3:30 PM

Also, any ETA for this?

CChinoman10 Also, any ETA for this?

yevgen•10/23/24, 8:48 AM

we will be aiming for mid Q1

yevgen•10/24/24, 2:34 PM

We have increased Vectorize limits for v2 as per https://developers.cloudflare.com/vectorize/platform/limits/

Cloudflare Docs

Limits | Vectorize

The following limits apply to accounts, indexes and vectors (as specified):

yanlin•10/25/24, 6:49 PM

For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ?

another question is that any upsert support that only updates the metadata based on id ? Whats the recommended way to update metadata?

freddw•10/26/24, 7:27 PM

Is best practice for supporting more complicated query semantics than just filtering to upsert vectors with a metadata field containing foreign keys to D1 records? Then implement your own pre/post filtering between the two?

freddw•10/26/24, 7:32 PM

as an aside it seems like if the reasoning behind launching vectorize as a separate product from D1 is that either sqlite did not support vector search at the time of launch, or that it was not considered an ideal substrate for vector search, it could be nice to integrate vectorize within D1 under the hood (with restrictions to what is possible within vectorize databases) so that users could write simpler queries/have easier setup/less to manage. Because at the end of the day a lot of people (like me) will want to combine topK with joins and it'd be nice to not have to implement some of that myself, assuming I'm not missing something re: the current impl

yanlin•10/28/24, 12:22 AM

Given the limitations of the meta-filtering operation, I have to leverage a nested key structure as documented:
{ "pandas.nice": 42 }
// looks for { "pandas": { "nice": 42 } }

However, when I attempt to upsert metadata where the key contains a JSON object, as specified in the documentation, I encounter the following error:

VECTOR_UPSERT_ERROR (code = 40019): invalid metadata for vector id=“jhbqpwft0qogbr1fs6re-0”; the metadata JSON object cannot contain JSON objects.

I’m not sure what to do next. Does anyone have any suggestions? Thanks!

JJerome Hi there! Now that V2 has GA'd, we will disable the **creation** of V1 indexes d...

Talljoe•10/28/24, 6:56 AM

When do you expect to update the Typescript API to use V2? I'm running into a problem where Wrangler doesn't support the --deprecated-v1--deprecated-v1 flag on certain calls. I just created the index today so I'm very happy to use V2 but don't want to roll my own client.

npm

cloudflare

The official TypeScript library for the Cloudflare API. Latest version: 3.5.0, last published: 3 months ago. Start using cloudflare in your project by running npm i cloudflarenpm i cloudflare. There are 93 other projects in the npm registry using cloudflare.

Yyanlin For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? anoth...

yevgen•10/28/24, 11:28 AM

For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ?

Yes

Yyanlin For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? anoth...

yevgen•10/28/24, 11:29 AM

another question is that any upsert support that only updates the metadata based on id ? Whats the recommended way to update metadata?

It's on a roadmap to provide

Jerome•10/28/24, 12:15 PM

We posted a deep dive into how we are building Vectorize on Cloudflare's blog. https://blog.cloudflare.com/building-vectorize-a-distributed-vector-database-on-cloudflare-developer-platform/ Check it out

The Cloudflare Blog

Building Vectorize, a distributed vector database, on Cloudflare’s ...

Vectorize was recently upgraded and made generally available, now supporting indexes of up to 5 million vectors, delivering faster responses, with lower pricing and a free tier. This post dives deep into how we built Vectorize to enable these improvements.

Yyevgen > For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? Yes

yanlin•10/28/24, 7:26 PM

thanks. Seems not null is not functional, for example, for any not indexed properties , if i use {"$ne","null"}, it always return everything

Yyanlin thanks. Seems not null is not functional, for example, for any not indexed prope...

Isaac McFadyen•10/28/24, 9:21 PM

Are you quoting nullnull like that?

Isaac McFadyen•10/28/24, 9:21 PM

Because that's a string "null""null", not actual null.

Isaac McFadyen•10/28/24, 9:22 PM

Whereas your initial question wasn't quoted, meaning it would be actual nullnull

IIsaac McFadyen Whereas your initial question wasn't quoted, meaning it would be actual `null`

yanlin•10/28/24, 11:24 PM

sorry for the typo i did {"path": {"$ne": null}, } and the path prop is not indexed

Rahul gangotri•10/29/24, 7:13 AM

Quick question

Why do we need to create metadata index and even if we have to create, why only 1 key at a time and upto 10 index is the limit? While upstash allows to filter on metadata normally on all metadata
Can we also expect glob and more type of searches operators along with eq and neq

Like : contains , IN, not in, glob, pattern match, and , or etc

can we also have hybrid search like a cosine similarity search with Full text search as well for better search in future?

leesus•10/29/24, 10:15 AM

Any updates/timeline on the automatic migration tool from v1 to v2 databases?

yevgen•10/30/24, 11:20 AM

Vectorize now has a significantly improved query latency through REST API https://developers.cloudflare.com/vectorize/platform/changelog/#2024-10-28

Cloudflare Docs

Changelog | Vectorize

EndiMOP•10/30/24, 9:07 PM

Hey! Whats the limit on insert/upsert.
for some reasons, my vector inserts/upsers have stoped at 128.04k vectors.
In dev, in another vectorize database, everything is working fine. Is there any hidden limit we dont know about???

EndiMOP•10/30/24, 9:10 PM

@yevgen

Yota•10/30/24, 9:25 PM

Is there a way to dynamically add Vectorize bindings programatically?

YYota Is there a way to dynamically add Vectorize bindings programatically?

Yota•10/30/24, 9:27 PM

ah i found it https://developers.cloudflare.com/vectorize/best-practices/create-indexes/

Yota•10/30/24, 9:30 PM

Hmmm, but these don't seem to be enough number for my use-case
Indexes per account 50,000 (Workers Paid)

EndiMOP•10/31/24, 12:26 AM

This issue is happening from : 2024-10-25T14:36:55.873Z
whatever improvement was made on this date, caused this to break.
Triggering it from binding or api, even tho the call return succees true, it actually does nothing on the index. No delete, no insert, no upset

EEndiM Hey! Whats the limit on insert/upsert. for some reasons, my vector inserts/upse...

yevgen•10/31/24, 8:52 AM

Nothing should have caused such issues, let me check with the team, it could be delayed as most of the team is in US time zone.

Yyevgen Nothing should have caused such issues, let me check with the team, it could be ...

EndiMOP•10/31/24, 8:52 AM

Thanks for getting back! Okay thank you.

yevgen•11/1/24, 1:16 PM

Vectorize update:

2024-10-30: Vectorize has local mode support in Wrangler @3.84.0 https://github.com/cloudflare/workers-sdk/releases/tag/wrangler%403.84.0

GitHub

Release wrangler@3.84.0 · cloudflare/workers-sdk

Minor Changes

#6999 0111edb Thanks @garvit-gupta! - docs: Vectorize GA Announcement Banner

#6916 a33a133 Thanks @garrettgu10! - Local development now supports Vectorize bindings

#7004 15ef013...

Samko•11/1/24, 2:53 PM

api idea - It would be great to have method for get/delete (all)vectors by namespacenamespace mainly delete use-case would be very helpful - in my case I have namespace for every "project" and I would like an easy way to clean-up.

ac•11/1/24, 9:09 PM

I am noticing a large number of AiError: 3001: Unknown internal errorAiError: 3001: Unknown internal error s when doing some Vectorize queries. Is there any way to get more insight into what these are? I have no clue if it's not finding a vector by ID, if I'm getting rate limited, or if it's something I can't control

thousandmiles•11/5/24, 1:23 AM

I've been browsing through messages in the past and documentations on vectorize, looks like now it's a great time to jump in, considering the beta release of workflow as well. though my concern is full RAG pipeline still needs a long way to go, so I'd prefer develop locally using llamaindex+chromadb(or equivalent)+litellm(embedding, inference, rerank, whisper, etc) first, before switching to cloudflare edge side completely, which I do need at a certain point because the product I am building requires global coverage.

may I ask guy who already had previous experiences here, that, is this a viable plan, or anything I should keep in mind for choosing tech stack or develop or doing gitops? thanks a million!

Rubi•11/7/24, 8:44 AM

Hi, is Cloudflare Vectorize now support $in metadata filter ?

yevgen•11/13/24, 8:19 PM

Vectorize update:
Added support for $in$in and $nin$nin metadata filters
https://developers.cloudflare.com/vectorize/platform/changelog/

Cloudflare Docs

Changelog | Vectorize

leesus•11/15/24, 11:30 AM

Is supporting arrays with metadata filtering on the roadmap? I'm migrating a v1 index to v2, and looking at adding a metadata index while I'm at it, but my metadata has string array properties. According to the docs this isn't possible now, but will it be in the near future? If not, I'll need to rethink how my metadata is stored before the migration.

CedricHadjian•11/18/24, 4:43 PM

How can I see what vectors I have in my index? I checked the commands I could run, and none mentioned showing the vectors I got in my index. I want to be able to delete and test stuff before I run the actual thing. Any guidance would be appreciated.

leesus•11/21/24, 1:27 PM

Are there any tips and tricks to improve query performance against Vectorize?

We make use of (per-project) namespaces for limiting the scope of similarity searches, and are seeing very slow performance on initial query against a namespace, and subsequent queries are much faster. One example has roughly 100 vectors and is taking ~3s for first query vs ~150ms for subsequent queries. Are cold-starts a thing with Vectorize?

momegas•11/22/24, 5:44 PM

Hello, Quick question, I get really slow response times in Vectorise. Like seconds of response time. My code is very simple, so nothing else should be slowing it down. I am in Greece btw, so not sure if that plays a role. I get the same speeds when deploying the worker.

  }),
  async (c) => {
    const text = "XXX";
    const e = await embedText(c, text);
    const r = await c.env.VECTORIZE.query(e, {
      topK: 5,
      returnValues: false,
      returnMetadata: "none",
    });

    return c.json({ matches: r.matches });
  }
);

  }),
  async (c) => {
    const text = "XXX";
    const e = await embedText(c, text);
    const r = await c.env.VECTORIZE.query(e, {
      topK: 5,
      returnValues: false,
      returnMetadata: "none",
    });

    return c.json({ matches: r.matches });
  }
);

Isaac McFadyen•11/27/24, 12:06 AM

@Kingsley Michael I removed your message as this is the channel for #vectorize, please use #off-topic for anything unrelated to Cloudflare products

Also, one other question. On the filter operation there is only eq and ne. Is there a way to use con

Similar Threads

Similar Threads

Similar Threads