is this approach is good of create a unique NS for each user i have and keep their vectors in that n

is this approach is good of create a unique NS for each user i have and keep their vectors in that namespace?
and multiple indexes for set of users

why is it onnly limited to 100 indexes per account, can we request to increase this?

Rahul gangotriOP•10/21/24, 3:27 PM

the product i am planning is going to store a lot of vectors

Rahul gangotriOP•10/21/24, 3:28 PM

we are already managing huge number of vectors using qdrant self hosted, but looking for cloudflare solution

Rahul gangotriOP•10/21/24, 3:32 PM

anyone?

RRahul gangotri is this approach is good of create a unique NS for each user i have and keep the...

yevgen•10/22/24, 12:01 PM

We are increasing limits (# of indexes per account and # of namespaces per index) for Workers Paid plan this week, stay tuned.

Rahul gangotriOP•10/22/24, 12:02 PM

RRahul gangotri is this approach is good of create a unique NS for each user i have and keep the...

Chinoman10•10/22/24, 3:15 PM

If users don't need to access each other's vectors (if they should be silo'ed), then yes, creating a namespace per user is OK, although right-now there's a 1k namespaces limit per index.
If they should be able to share them, then they should share a namespace and you use metadata filters instead for user segregation.

So at the end of the day, you'll have to think what suits your use-case the best.

Yyevgen there is no support for contains. We are planning add support for the following ...

Chinoman10•10/22/24, 3:30 PM

Also, any ETA for this?

CChinoman10 Also, any ETA for this?

yevgen•10/23/24, 8:48 AM

we will be aiming for mid Q1

yevgen•10/24/24, 2:34 PM

We have increased Vectorize limits for v2 as per https://developers.cloudflare.com/vectorize/platform/limits/

Cloudflare Docs

Limits | Vectorize

The following limits apply to accounts, indexes and vectors (as specified):

yanlin•10/25/24, 6:49 PM

For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ?

another question is that any upsert support that only updates the metadata based on id ? Whats the recommended way to update metadata?

freddw•10/26/24, 7:27 PM

Is best practice for supporting more complicated query semantics than just filtering to upsert vectors with a metadata field containing foreign keys to D1 records? Then implement your own pre/post filtering between the two?

freddw•10/26/24, 7:32 PM

as an aside it seems like if the reasoning behind launching vectorize as a separate product from D1 is that either sqlite did not support vector search at the time of launch, or that it was not considered an ideal substrate for vector search, it could be nice to integrate vectorize within D1 under the hood (with restrictions to what is possible within vectorize databases) so that users could write simpler queries/have easier setup/less to manage. Because at the end of the day a lot of people (like me) will want to combine topK with joins and it'd be nice to not have to implement some of that myself, assuming I'm not missing something re: the current impl

yanlin•10/28/24, 12:22 AM

Given the limitations of the meta-filtering operation, I have to leverage a nested key structure as documented:
{ "pandas.nice": 42 }
// looks for { "pandas": { "nice": 42 } }

However, when I attempt to upsert metadata where the key contains a JSON object, as specified in the documentation, I encounter the following error:

VECTOR_UPSERT_ERROR (code = 40019): invalid metadata for vector id=“jhbqpwft0qogbr1fs6re-0”; the metadata JSON object cannot contain JSON objects.

I’m not sure what to do next. Does anyone have any suggestions? Thanks!

JJerome Hi there! Now that V2 has GA'd, we will disable the **creation** of V1 indexes d...

Talljoe•10/28/24, 6:56 AM

When do you expect to update the Typescript API to use V2? I'm running into a problem where Wrangler doesn't support the

--deprecated-v1

--deprecated-v1

flag on certain calls. I just created the index today so I'm very happy to use V2 but don't want to roll my own client.

npm

cloudflare

The official TypeScript library for the Cloudflare API. Latest version: 3.5.0, last published: 3 months ago. Start using cloudflare in your project by running

npm i cloudflare

npm i cloudflare

. There are 93 other projects in the npm registry using cloudflare.

Yyanlin For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? anoth...

yevgen•10/28/24, 11:28 AM

For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ?

Yes

Yyanlin For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? anoth...

yevgen•10/28/24, 11:29 AM

another question is that any upsert support that only updates the metadata based on id ? Whats the recommended way to update metadata?

It's on a roadmap to provide

Jerome•10/28/24, 12:15 PM

We posted a deep dive into how we are building Vectorize on Cloudflare's blog. https://blog.cloudflare.com/building-vectorize-a-distributed-vector-database-on-cloudflare-developer-platform/ Check it out

The Cloudflare Blog

Building Vectorize, a distributed vector database, on Cloudflare’s ...

Vectorize was recently upgraded and made generally available, now supporting indexes of up to 5 million vectors, delivering faster responses, with lower pricing and a free tier. This post dives deep into how we built Vectorize to enable these improvements.

Yyevgen > For the metadata filtering, is it ok to do {"attribute":{"$ne", null} } ? Yes

yanlin•10/28/24, 7:26 PM

thanks. Seems not null is not functional, for example, for any not indexed properties , if i use {"$ne","null"}, it always return everything

Yyanlin thanks. Seems not null is not functional, for example, for any not indexed prope...

Isaac McFadyen•10/28/24, 9:21 PM

Are you quoting

null

null

like that?

Isaac McFadyen•10/28/24, 9:21 PM

Because that's a string

"null"

"null"

, not actual null.

Isaac McFadyen•10/28/24, 9:22 PM

Whereas your initial question wasn't quoted, meaning it would be actual

null

null

IIsaac McFadyen Whereas your initial question wasn't quoted, meaning it would be actual `null`

yanlin•10/28/24, 11:24 PM

sorry for the typo i did {"path": {"$ne": null}, } and the path prop is not indexed

Rahul gangotriOP•10/29/24, 7:13 AM

Quick question

Why do we need to create metadata index and even if we have to create, why only 1 key at a time and upto 10 index is the limit? While upstash allows to filter on metadata normally on all metadata
Can we also expect glob and more type of searches operators along with eq and neq

Like : contains , IN, not in, glob, pattern match, and , or etc

can we also have hybrid search like a cosine similarity search with Full text search as well for better search in future?

leesus•10/29/24, 10:15 AM

Any updates/timeline on the automatic migration tool from v1 to v2 databases?

yevgen•10/30/24, 11:20 AM

Vectorize now has a significantly improved query latency through REST API https://developers.cloudflare.com/vectorize/platform/changelog/#2024-10-28

Cloudflare Docs

Changelog | Vectorize

EndiM•10/30/24, 9:07 PM

Hey! Whats the limit on insert/upsert.
for some reasons, my vector inserts/upsers have stoped at 128.04k vectors.
In dev, in another vectorize database, everything is working fine. Is there any hidden limit we dont know about???

EndiM•10/30/24, 9:10 PM

@yevgen

Yota•10/30/24, 9:25 PM

Is there a way to dynamically add Vectorize bindings programatically?

YYota Is there a way to dynamically add Vectorize bindings programatically?

Yota•10/30/24, 9:27 PM

ah i found it https://developers.cloudflare.com/vectorize/best-practices/create-indexes/

Yota•10/30/24, 9:30 PM

Hmmm, but these don't seem to be enough number for my use-case
Indexes per account 50,000 (Workers Paid)

EndiM•10/31/24, 12:26 AM

This issue is happening from : 2024-10-25T14:36:55.873Z
whatever improvement was made on this date, caused this to break.
Triggering it from binding or api, even tho the call return succees true, it actually does nothing on the index. No delete, no insert, no upset

EEndiM Hey! Whats the limit on insert/upsert. for some reasons, my vector inserts/upse...

yevgen•10/31/24, 8:52 AM

Nothing should have caused such issues, let me check with the team, it could be delayed as most of the team is in US time zone.

Yyevgen Nothing should have caused such issues, let me check with the team, it could be ...

EndiM•10/31/24, 8:52 AM

Thanks for getting back! Okay thank you.

yevgen•11/1/24, 1:16 PM

Vectorize update:

2024-10-30: Vectorize has local mode support in Wrangler @3.84.0 https://github.com/cloudflare/workers-sdk/releases/tag/wrangler%403.84.0

GitHub

Release wrangler@3.84.0 · cloudflare/workers-sdk

Minor Changes

#6999 0111edb Thanks @garvit-gupta! - docs: Vectorize GA Announcement Banner

#6916 a33a133 Thanks @garrettgu10! - Local development now supports Vectorize bindings

#7004 15ef013...

Samko•11/1/24, 2:53 PM

api idea - It would be great to have method for get/delete (all)vectors by

namespace

namespace

mainly delete use-case would be very helpful - in my case I have namespace for every "project" and I would like an easy way to clean-up.

ac•11/1/24, 9:09 PM

I am noticing a large number of

AiError: 3001: Unknown internal error

AiError: 3001: Unknown internal error

s when doing some Vectorize queries. Is there any way to get more insight into what these are? I have no clue if it's not finding a vector by ID, if I'm getting rate limited, or if it's something I can't control

thousandmiles•11/5/24, 1:23 AM

I've been browsing through messages in the past and documentations on vectorize, looks like now it's a great time to jump in, considering the beta release of workflow as well. though my concern is full RAG pipeline still needs a long way to go, so I'd prefer develop locally using llamaindex+chromadb(or equivalent)+litellm(embedding, inference, rerank, whisper, etc) first, before switching to cloudflare edge side completely, which I do need at a certain point because the product I am building requires global coverage.

may I ask guy who already had previous experiences here, that, is this a viable plan, or anything I should keep in mind for choosing tech stack or develop or doing gitops? thanks a million!

Rubi•11/7/24, 8:44 AM

Hi, is Cloudflare Vectorize now support $in metadata filter ?

yevgen•11/13/24, 8:19 PM

Vectorize update:
Added support for

$in

$in

and

$nin

$nin

metadata filters
https://developers.cloudflare.com/vectorize/platform/changelog/

Cloudflare Docs

Changelog | Vectorize

leesus•11/15/24, 11:30 AM

Is supporting arrays with metadata filtering on the roadmap? I'm migrating a v1 index to v2, and looking at adding a metadata index while I'm at it, but my metadata has string array properties. According to the docs this isn't possible now, but will it be in the near future? If not, I'll need to rethink how my metadata is stored before the migration.

CedricHadjian•11/18/24, 4:43 PM

How can I see what vectors I have in my index? I checked the commands I could run, and none mentioned showing the vectors I got in my index. I want to be able to delete and test stuff before I run the actual thing. Any guidance would be appreciated.

leesus•11/21/24, 1:27 PM

Are there any tips and tricks to improve query performance against Vectorize?

We make use of (per-project) namespaces for limiting the scope of similarity searches, and are seeing very slow performance on initial query against a namespace, and subsequent queries are much faster. One example has roughly 100 vectors and is taking ~3s for first query vs ~150ms for subsequent queries. Are cold-starts a thing with Vectorize?

momegas•11/22/24, 5:44 PM

Hello, Quick question, I get really slow response times in Vectorise. Like seconds of response time. My code is very simple, so nothing else should be slowing it down. I am in Greece btw, so not sure if that plays a role. I get the same speeds when deploying the worker.

  }),
  async (c) => {
    const text = "XXX";
    const e = await embedText(c, text);
    const r = await c.env.VECTORIZE.query(e, {
      topK: 5,
      returnValues: false,
      returnMetadata: "none",
    });

    return c.json({ matches: r.matches });
  }
);

  }),
  async (c) => {
    const text = "XXX";
    const e = await embedText(c, text);
    const r = await c.env.VECTORIZE.query(e, {
      topK: 5,
      returnValues: false,
      returnMetadata: "none",
    });

    return c.json({ matches: r.matches });
  }
);

Isaac McFadyen•11/27/24, 12:06 AM

@Kingsley Michael I removed your message as this is the channel for #vectorize, please use #off-topic for anything unrelated to Cloudflare products

Alex Karp•11/27/24, 2:01 AM

The limits page says that there's a maximum of 5,000,000 vectors per index, but I'm getting a 4002 error when attempting to upsert past 240,000. any idea why or things I should change?

EDIT: RESOLVED, I was using V1 instead of V2

CedricHadjian•11/27/24, 9:08 PM

TypeError: vectors.map is not a function

What does this mean when it's triggered from this:
await vectorIndex.insert({
id: String(data[0][dataId]),
values: vectorArray,
metadata
});

EDIT:
I tested it on a small function:

            if (request.method === 'GET' && pathname === `/embed`) {
                try {
                    const text = "Liam Marshall"

                    const response = await env.AI.run("@cf/baai/bge-small-en-v1.5", {
                        text,
                    });
    
                    await env.TESTVECTORS.insert({
                        id: 1,
                        values: response.data[0],
                        metadata: {
                            key: "value"
                        }
                    })
                    
                    return new Response(JSON.stringify(response), {
                        headers: {
                            'content-type': 'text/plain;charset=UTF-8',
                            'Access-Control-Allow-Origin': '*',
                        },
                    });
                } catch (error) {
                    console.log(error);
                }

            }

            if (request.method === 'GET' && pathname === `/embed`) {
                try {
                    const text = "Liam Marshall"

                    const response = await env.AI.run("@cf/baai/bge-small-en-v1.5", {
                        text,
                    });
    
                    await env.TESTVECTORS.insert({
                        id: 1,
                        values: response.data[0],
                        metadata: {
                            key: "value"
                        }
                    })
                    
                    return new Response(JSON.stringify(response), {
                        headers: {
                            'content-type': 'text/plain;charset=UTF-8',
                            'Access-Control-Allow-Origin': '*',
                        },
                    });
                } catch (error) {
                    console.log(error);
                }

            }

I got the same error: vectors.map is not a function...

CCedricHadjian TypeError: vectors.map is not a function What does this mean when it's triggere...

CedricHadjian•11/27/24, 10:25 PM

@yevgen Any help would be appreciated, we're a high tier client and we want to implement vectorize over our app. But we're facing vectors.map is not a function on simple calls as well.

CCedricHadjian @yevgen Any help would be appreciated, we're a high tier client and we want to i...

ggu•11/29/24, 6:57 PM

Does this happen just locally or also when deployed?

}), async (c) => { const text = "XXX"; const e = await embedText(c, text); const r = await c.env.VECTORIZE.query(e, { topK: 5, returnValues: false, returnMetadata: "none", }); return c.json({ matches: r.matches }); } );

if (request.method === 'GET' && pathname === `/embed`) { try { const text = "Liam Marshall" const response = await env.AI.run("@cf/baai/bge-small-en-v1.5", { text, }); await env.TESTVECTORS.insert({ id: 1, values: response.data[0], metadata: { key: "value" } }) return new Response(JSON.stringify(response), { headers: { 'content-type': 'text/plain;charset=UTF-8', 'Access-Control-Allow-Origin': '*', }, }); } catch (error) { console.log(error); } }

is this approach is good of create a unique NS for each user i have and keep their vectors in that n

Similar Threads

Similar Threads

Similar Threads