Would love to be able to dynamically create & bind an index per customer/tenant. Is that something t

Would love to be able to dynamically create & bind an index per customer/tenant. Is that something that’s on the roadmap?

Yyevgen So far we haven't observed anyone using Vectorize to encounter our current limit...

hannojg•9/8/24, 1:24 AM

In our use case we don’t have a lot of data belonging to one category, but a lot of smaller data belonging to many segments.
So I am wondering how I can segment the vectors. The options I see are:
- creating an index per segment
- using namespaces in the same index
- metadata filtering (less preferred)

Both indexes and namespaces have quite a tight limit for our use case

Ccloudkite Would love to be able to dynamically create & bind an index per customer/tenant....

hannojg•9/8/24, 1:27 AM

Kind of like this - if we already have 20.000 customers, and each customer has ~15 projects for which I’d need separate segments, it’s hard to do that with the current limits

yevgen•9/9/24, 8:45 AM

I think metadata filtering is the best way forward. If you prefer using namespace for whaterever reason (nothing special other then we index it by default) we can lift # of namespaces limit for your account.

yevgen•9/9/24, 8:48 AM

Creating an index per segment could be an option as long as your index size will be at least 50k vectors, otherwise one index for all data is probably a better match.

Ccloudkite Would love to be able to dynamically create & bind an index per customer/tenant....

yevgen•9/9/24, 8:53 AM

yes, Q1 2025 we plan to ship support for Vectorize dynamic bindings.

cosbgn•9/9/24, 1:26 PM

Can someone give me a brief guide/bullet points on how to use vectorize for a saas?

- Should I create a namespace per user and query that namespace?
- Should I just add them all in one namespace and query for metatags?

What's the best proactice?

cosbgn•9/9/24, 1:28 PM

1000 namespaces for now it would be enough to have one per user, but that can eventually be increased, right?

cosbgn•9/9/24, 1:29 PM

Also, is there any restriction on saving the original text in the metadata? It would save me an extra call to D1

cosbgn•9/9/24, 1:30 PM

10KiB of metadata is enough to store most text

cosbgn•9/9/24, 1:31 PM

all of it actually + some ids and some other small info

Ccosbgn 1000 namespaces for now it would be enough to have one per user, but that can ev...

yevgen•9/9/24, 5:32 PM

yes, it could be increased.

ac•9/11/24, 2:03 PM

Hi, when trying to get vector values the Vectorize API docs say

values

values

is "an array of number, Float32Array, or Float64Array as the vector embedding itself."

Can someone tell me what actually determines which one of these types the vector is?

Aac Hi, when trying to get vector values the Vectorize API docs say `values` is "an ...

Jerome•9/11/24, 5:57 PM

Hi! I believe you're refering to https://developers.cloudflare.com/vectorize/best-practices/query-vectors/; the format you use to query your DB through a binding can be either of these, it'll be interpreted the same way, as a floating point number array. You also don't have to use the type that matches the one you used when inserting/upserting, all number formats are interoperable for insert/upsert and query.

JJerome Hi! I believe you're refering to https://developers.cloudflare.com/vectorize/bes...

ac•9/11/24, 8:31 PM

Thanks, but I'm talking about the return value. It seems like there are instances when it is Float32Array or Float64Array instead of number[], but I'm trying to determine how to know which it will be.

ac•9/13/24, 8:53 PM

I am getting 504 Gateway Timeouts intermittently when running

wrangler vectorize list

wrangler vectorize list

. Is there an ongoing incident?

ac•9/13/24, 8:55 PM

Also getting this intermittently on deploys, which has never happened before:

Error: Failed to publish your Function. Got error: binding <vectorize index name> of type vectorize failed to generate: internal error; please try again later or contact support

Error: Failed to publish your Function. Got error: binding <vectorize index name> of type vectorize failed to generate: internal error; please try again later or contact support

Aac I am getting 504 Gateway Timeouts intermittently when running `wrangler vectoriz...

yevgen•9/15/24, 12:42 PM

We saw intermittent issues with one of the service instances running in our Core datacenter. It has resolved now.

cosbgn•9/15/24, 4:07 PM

Any way to filter metadata as array includes?

In my use case I have some files and they have tags like ["books", "work"], etc these tags are set by the users and I don't know them in advance.

Ideally I would be able to query embeddings only for "work" or only for "books" ]

cosbgn•9/16/24, 8:51 AM

Any way to delete a namespace and all the vectors inside it via rest API? doesn't seem to be documented

charl.dev•9/16/24, 8:32 PM

Is setting a ttl on a vector entry, similar to kv on the cards?

Have items I want to clear out like 3 months after creation. Can do like a nightly purge but would be a great feature IMO

Ccosbgn Any way to delete a namespace and all the vectors inside it via rest API? doesn'...

Sunny•9/17/24, 5:04 PM

I've been deleting through "wrangler vectorize delete <index_name>"

Sunny•9/17/24, 5:04 PM

though I'm running into an issue when trying to create a new index under the same name

Sunny•9/17/24, 5:05 PM

it gets deleted off my dashboard but I get this error when creating a duplicate index. Not sure how to fully remove it and its history

Sunny•9/17/24, 5:06 PM

🚧 Creating index: 'database-name'

✘ [ERROR] A request to the Cloudflare API (/accounts/b7b5a2f84f22862a502266ecc6abb2a0/vectorize/v2/indexes) failed.

  vectorize.index.deleted - Index name "database-name" [code: 3005]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

🚧 Creating index: 'database-name'

✘ [ERROR] A request to the Cloudflare API (/accounts/b7b5a2f84f22862a502266ecc6abb2a0/vectorize/v2/indexes) failed.

  vectorize.index.deleted - Index name "database-name" [code: 3005]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

chronark•9/18/24, 8:47 AM

is there a way to clear data in an index in the UI?
I can't see any and the docs aren't helping either

Sunny•9/18/24, 3:28 PM

I think you can only delete elements through the wrangler cli, and it’s only element by element, reference by id

Sunny•9/18/24, 3:28 PM

Otherwise, again through cli, you delete the whole index and make a new one

matt•9/19/24, 7:15 PM

Any roadmap for when max dimensions will be increased? I'd love to use text-embedding-3-large but it's 3072 dimensions.

Mmatt Any roadmap for when max dimensions will be increased? I'd love to use text-embe...

thipperz•9/20/24, 12:13 AM

you know that in the meantime you can specify 1536 dimensions in the open ai api call right?

matt•9/20/24, 3:25 AM

Yes I’m aware thank you. I’d like to take full advantage of it though.

mikibo•9/20/24, 5:51 PM

Hello there! Looking for how to use logical OR in vectorize metadata filtering. Is it possible? I've seen

AND

AND

(implicit) but can't find

OR

OR

charl.dev•9/20/24, 7:30 PM

Is vectorize available in DOs? I’ve bound it via toml and via env type but when I log this.env, I can see all my other bindings but not vectorize.

When I run wrangler dev, it does say the worker has access to vectorize

Ccharl.dev Is vectorize available in DOs? I’ve bound it via toml and via env type but when ...

charl.dev•9/20/24, 7:32 PM

Nvm was running remote flag with npm run dev lol

Eric•9/25/24, 4:24 AM

Hi all, when I delete an index, and try to create it with the same name through the API, I get

{
    "result": null,
    "success": false,
    "errors": [{
        "code": 3005,
        "message": "vectorize.index.deleted - Index name \"your_index_name\""
    }],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [{
        "code": 3005,
        "message": "vectorize.index.deleted - Index name \"your_index_name\""
    }],
    "messages": []
}

Is that by design? Is there a timelimit that it'll be purged on the backend?

Eric•9/25/24, 4:28 AM

Also note that I can't find anything related to this error message in the docs.

Eric•9/25/24, 4:36 AM

@Jerome (tagging because you seem familiar with this)

EEric Hi all, when I delete an index, and try to create it with the same name through ...

Jerome•9/25/24, 5:08 AM

Hi there! Yes index names are single use, cannot reuse after deletion. We are working on the documentation to clarify this

JJerome Hi there! Yes index names are single use, cannot reuse after deletion. We are wo...

Eric•9/25/24, 6:53 AM

Gotcha. I created the index

test

test

and deleted it. Does that mean I can never create an index called

test

test

for the life of my account?

Eric•9/25/24, 7:05 AM

Also, say I want to have multiple collections of data be indexed, but such data is completely unrelated to each other. Is it better to:
1. Create a single index and have it search the embeddings by namespace? (Also: Is the intended usage for namespaces specifically for file/data types?)
2. Store grouping information in the metadata, and have it query by the metadata? I doubt this one, since the collections are intended to be completely unrelated.
3. Create new indexes for each collection of data?

EEric Hi all, when I delete an index, and try to create it with the same name through ...

Eric•9/25/24, 7:06 AM

@nullptr

EEric Gotcha. I created the index `test` and deleted it. Does that mean I can never cr...

Jerome•9/25/24, 9:23 AM

Yes

EEric Also, say I want to have multiple collections of data be indexed, but such data ...

Jerome•9/25/24, 9:34 AM

1 index per collection seems the way to go if you will not want to run queries spanning several collections, and the number of collections is small and constant (as you'll want to bind against them in your worker, which is a manual operation).

Otherwise, one index with logic partitioning of the collections; namespace is the designated way to achieve this (but metadata would also work as you pointed out). With this approach (1 index with logic partitioning) you need to make sure however that your collections all use vectors that are the same dimension, indexed and queried with the same distance metric, and ideally from the same source (the same LLM for instance) to ensure consistency in the vector data, for best indexing accuracy.

EEric Hi all, when I delete an index, and try to create it with the same name through ...

Rihan•9/25/24, 12:45 PM

I've just ran into this limitation today too

JJerome Hi there! Yes index names are single use, cannot reuse after deletion. We are wo...

Rihan•9/25/24, 1:10 PM

Is there a planned way to empty a Vectorize index? My use case is resetting a preview environment's index. If not, the only way forward I see is recreating with a random string suffix.

Rihan•9/25/24, 3:03 PM

Furthermore would it be possible to support name reuse in the future?

RRihan Furthermore would it be possible to support name reuse in the future?

Jerome•9/25/24, 3:42 PM

Yes, we are aware this is an inconvenience and supporting index name reuse after deletion is on the roadmap, no shipping date yet

Eric•9/25/24, 11:25 PM

Gotcha. Thanks for the info!

Eric•9/26/24, 3:32 AM

A little follow up; I still don't quite understand the design for multi-collection dynamic data storage. Binding an index to a worker is a manual operation so dynamically created indexes aren't possible. This infers that having multiple indexes exist only for the purpose of different dimension sizes/metrics/embedding functions (since these are determined at creation time, and are constant) OR unless the data stored is also constant.

Say I have a user in my application with his/her own collection of data and they want to put it into an index dynamically. This must be done within a single index that all users share (separated by namespace for each user). Looking at the limits here:
https://developers.cloudflare.com/vectorize/platform/limits/
"Maximum vectors per index" shows a max of 5mil for the beta. Say the limit grows to 100mil- although 100mil embeddings is a lot for a single user, this won't scale well if I enable my users to upload data, forced to upload to a single index.

The (current beta) limit for number of indexes is 100. Say in the future we're able to bind indexes to workers dynamically. That still won't suffice for a user base even if that number grows to 10k or more.

In order to properly handle dynamic scaling, I could create all 100 possible indexes now, all empty, and use them as overflow space once the current index fills up. Coding this containerization algo would suck and I doubt that's the intended purpose.

I get the impression that uploading data and initializing indexes is a constant operation. I'm designing an application now so any advice on how to properly design my architecture for the future would help! @Jerome

🚧 Creating index: 'database-name' ✘ [ERROR] A request to the Cloudflare API (/accounts/b7b5a2f84f22862a502266ecc6abb2a0/vectorize/v2/indexes) failed. vectorize.index.deleted - Index name "database-name" [code: 3005] If you think this is a bug, please open an issue at: https://github.com/cloudflare/workers-sdk/issues/new/choose

Would love to be able to dynamically create & bind an index per customer/tenant. Is that something t

Similar Threads

Similar Threads

Similar Threads