Cloudflare Developers•16mo ago

Any roadmap for when max dimensions will be increased? I'd love to use text-embedding-3-large but it

Any roadmap for when max dimensions will be increased? I'd love to use text-embedding-3-large but it's 3072 dimensions.

Mmatt Any roadmap for when max dimensions will be increased? I'd love to use text-embe...

thipperz•9/20/24, 12:13 AM

you know that in the meantime you can specify 1536 dimensions in the open ai api call right?

mattOP•9/20/24, 3:25 AM

Yes I’m aware thank you. I’d like to take full advantage of it though.

mikibo•9/20/24, 5:51 PM

Hello there! Looking for how to use logical OR in vectorize metadata filtering. Is it possible? I've seen ANDAND (implicit) but can't find OROR.

charl.dev•9/20/24, 7:30 PM

Is vectorize available in DOs? I’ve bound it via toml and via env type but when I log this.env, I can see all my other bindings but not vectorize.

When I run wrangler dev, it does say the worker has access to vectorize

Ccharl.dev Is vectorize available in DOs? I’ve bound it via toml and via env type but when ...

charl.dev•9/20/24, 7:32 PM

Nvm was running remote flag with npm run dev lol

Eric•9/25/24, 4:24 AM

Hi all, when I delete an index, and try to create it with the same name through the API, I get

{
    "result": null,
    "success": false,
    "errors": [{
        "code": 3005,
        "message": "vectorize.index.deleted - Index name \"your_index_name\""
    }],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [{
        "code": 3005,
        "message": "vectorize.index.deleted - Index name \"your_index_name\""
    }],
    "messages": []
}

Is that by design? Is there a timelimit that it'll be purged on the backend?

Eric•9/25/24, 4:28 AM

Also note that I can't find anything related to this error message in the docs.

Eric•9/25/24, 4:36 AM

@Jerome (tagging because you seem familiar with this)

EEric Hi all, when I delete an index, and try to create it with the same name through ...

Jerome•9/25/24, 5:08 AM

Hi there! Yes index names are single use, cannot reuse after deletion. We are working on the documentation to clarify this

JJerome Hi there! Yes index names are single use, cannot reuse after deletion. We are wo...

Eric•9/25/24, 6:53 AM

Gotcha. I created the index testtest and deleted it. Does that mean I can never create an index called testtest for the life of my account?

Eric•9/25/24, 7:05 AM

Also, say I want to have multiple collections of data be indexed, but such data is completely unrelated to each other. Is it better to:

Create a single index and have it search the embeddings by namespace? (Also: Is the intended usage for namespaces specifically for file/data types?)
Store grouping information in the metadata, and have it query by the metadata? I doubt this one, since the collections are intended to be completely unrelated.
Create new indexes for each collection of data?

EEric Hi all, when I delete an index, and try to create it with the same name through ...

Eric•9/25/24, 7:06 AM

@nullptr

EEric Gotcha. I created the index `test` and deleted it. Does that mean I can never cr...

Jerome•9/25/24, 9:23 AM

Yes

EEric Also, say I want to have multiple collections of data be indexed, but such data ...

Jerome•9/25/24, 9:34 AM

1 index per collection seems the way to go if you will not want to run queries spanning several collections, and the number of collections is small and constant (as you'll want to bind against them in your worker, which is a manual operation).

Otherwise, one index with logic partitioning of the collections; namespace is the designated way to achieve this (but metadata would also work as you pointed out). With this approach (1 index with logic partitioning) you need to make sure however that your collections all use vectors that are the same dimension, indexed and queried with the same distance metric, and ideally from the same source (the same LLM for instance) to ensure consistency in the vector data, for best indexing accuracy.

EEric Hi all, when I delete an index, and try to create it with the same name through ...

Rihan•9/25/24, 12:45 PM

I've just ran into this limitation today too

JJerome Hi there! Yes index names are single use, cannot reuse after deletion. We are wo...

Rihan•9/25/24, 1:10 PM

Is there a planned way to empty a Vectorize index? My use case is resetting a preview environment's index. If not, the only way forward I see is recreating with a random string suffix.

Rihan•9/25/24, 3:03 PM

Furthermore would it be possible to support name reuse in the future?

RRihan Furthermore would it be possible to support name reuse in the future?

Jerome•9/25/24, 3:42 PM

Yes, we are aware this is an inconvenience and supporting index name reuse after deletion is on the roadmap, no shipping date yet

Eric•9/25/24, 11:25 PM

Gotcha. Thanks for the info!

Eric•9/26/24, 3:32 AM

A little follow up; I still don't quite understand the design for multi-collection dynamic data storage. Binding an index to a worker is a manual operation so dynamically created indexes aren't possible. This infers that having multiple indexes exist only for the purpose of different dimension sizes/metrics/embedding functions (since these are determined at creation time, and are constant) OR unless the data stored is also constant.

Say I have a user in my application with his/her own collection of data and they want to put it into an index dynamically. This must be done within a single index that all users share (separated by namespace for each user). Looking at the limits here:
https://developers.cloudflare.com/vectorize/platform/limits/
"Maximum vectors per index" shows a max of 5mil for the beta. Say the limit grows to 100mil- although 100mil embeddings is a lot for a single user, this won't scale well if I enable my users to upload data, forced to upload to a single index.

The (current beta) limit for number of indexes is 100. Say in the future we're able to bind indexes to workers dynamically. That still won't suffice for a user base even if that number grows to 10k or more.

In order to properly handle dynamic scaling, I could create all 100 possible indexes now, all empty, and use them as overflow space once the current index fills up. Coding this containerization algo would suck and I doubt that's the intended purpose.

I get the impression that uploading data and initializing indexes is a constant operation. I'm designing an application now so any advice on how to properly design my architecture for the future would help! @Jerome

Eric•9/26/24, 3:37 AM

Currently I'll be limited by:

100 users max where each index name can only be used once OR
5 mil embeddings total for the application OR
My coding ability to whip up a containerization algo, then limited by 100 indexes * 5mil vectors (in beta i know, but still)

I guess my question boils down to- how should I design my application such that it scales dynamically once beta is over? Thank you for being informative so far

Yyevgen yes, Q1 2025 we plan to ship support for Vectorize dynamic bindings.

Rihan•9/26/24, 9:49 AM

@Eric Vectorize dynamic bindings are coming hopefully in Q1. I believe Cloudflare can increase the number of indexes limit on an account basis too.

EEric Currently I'll be limited by: - 100 users max where each index name can only be ...

Akanbi•9/26/24, 5:43 PM

Cindy thud

AAkanbi Cindy thud

Eric•9/27/24, 12:44 AM

Who's Chindy thud?

Rubi•9/27/24, 8:11 AM

Is there any way to export Vectorize data? I want to migrate to another provider

Ccfnathan This is a symptom of how we have the beta deployed. As mentioned in the above th...

Rubi•9/27/24, 8:46 AM

hi, is the new deployed?

uri•9/29/24, 1:41 PM

hi, there is an option to backup and restore indexes?

uri•9/29/24, 1:43 PM

also, vectorize limit to the number of indexes one can create?

uri•9/29/24, 2:15 PM

I noticed on this page "https://developers.cloudflare.com/vectorize/platform/limits/" that the current limits are 100 indexes and 1,000 namespaces. While this might be sufficient for testing, it's far from enough for production use. Are there any plans to scale these limits—at least by a couple of orders of magnitude? Something closer to 1,000,000 or more would be ideal.

uri•9/29/24, 2:26 PM

I want to building my SaaS around 'Vectorize' as I really like the platform. However, with the current limitations in place, if these are expected to remain for the foreseeable future, I won't be able to use it for my needs. Additionally, dedicated functions or tools for backup and restore are essential for anyone looking to use this platform in production environments. Could someone provide insight into the current roadmap for 'Vectorize' in terms of limitations and development? What can we expect in the near future?

Rihan•10/2/24, 4:56 PM

With terminology, are Vectorize indexes and databases exchangeable terms, or are indexes the preferred way to refer to them?

Rihan•10/2/24, 4:56 PM

This is on Pages -> Settings -> Bindings -> Add -> Vectorize database

Ashley•10/2/24, 9:09 PM

Is the billing page broken? I clicked view next to monthly usage on the Vectorize dashboard and I get an oops page

echoes221•10/3/24, 10:56 PM

Remix + Cloudflare - running wrangler types generates the correct vectorize types, however the proxy stub doesn't exist in the loader (cloudflareDevProxyVitePlugin). Is this due to beta?

  const {
    params,
    context: { cloudflare },
  } = args;
 
  if (!cloudflare.env.VECTORIZE) { // <-- doesn't exist
    console.log("no key"); 
    return;
  }

  const {
    params,
    context: { cloudflare },
  } = args;
 
  if (!cloudflare.env.VECTORIZE) { // <-- doesn't exist
    console.log("no key"); 
    return;
  }

sathoro•10/4/24, 8:20 AM

can the 1,000 namespaces per index limit be easily increased? we use Pinecone but was looking at using Vectorize for a new feature and this limit is really putting us off

Eechoes221 Remix + Cloudflare - running wrangler types generates the correct vectorize type...

Rubi•10/4/24, 11:42 AM

yes, vectorize is currently not available on local

RRubi yes, vectorize is currently not available on local

echoes221•10/4/24, 2:31 PM

Thanks for the update!

Jerome•10/4/24, 3:41 PM

We're currently working on supporting local binding for vectorize, hopefully completed this month :soontm:

JJerome We're currently working on supporting local binding for vectorize, hopefully com...

echoes221•10/5/24, 2:12 PM

That would be great, thank you. In the meantime I’m generating embeddings using workers ai, then storing them in pg vector / querying in Postgres, seems to work well enough!

uri•10/6/24, 8:48 AM

what is "local binding for vectorize" mean?

Uuri what is "local binding for vectorize" mean?

Ashley•10/6/24, 9:46 AM

For a lot of Cloudflare products, you can use a similar version of the product locally. So when you run “wrangler dev”, the binding still works, so you can run/test locally - for Vectorize, that isn’t available yet, so you have to run it remote to test it (either deploy, or wrangler dev remote)

crunchy•10/6/24, 11:08 PM

Hi is there an additional information on the eventual consistency behaviour, can I assume the vectors are available for search in order? does the mutation identifier have an meaning? can it be used as a high water mark? I have a use case were a want to group vectors, it's tricky to do without knowing when vectors will be availible? presumable without a hwm I'd have to use a queue with some kind of back off but that won't help if the vectors are not indexed in order. presumble then the only solution would be repair on each read but i'm capped to 100 nearest neighbours.

Ccrunchy Hi is there an additional information on the eventual consistency behaviour, can...

Jerome•10/7/24, 6:53 AM

Hi! The DB state is eventually consistent, and all mutations (upsert, insert, delete, create metadata index, ...) are processed in the strict order they were given to the API. This means the index state is always reflecting all the mutations that were given to the API up to the last applied mutation, processed in order.
The mutationId acts as a high watermark indeed, you can compare the mutationId returned by any mutation operation and the one you get by calling https://developers.cloudflare.com/vectorize/reference/client-api/#get-index-info ; this will return the vector count, the last applied mutationId (again, in the sequence ordered as provided to the API) and the last UTC datetime this mutation corresponds to (useful if you don't keep track of the mutationIds)

vvotekeb•10/10/24, 12:33 PM

Does anyone know of any methods to create a backup for an index? I'm looking for a script/code that can pull all the data from one index and insert it to another.

kingmesal•10/10/24, 11:30 PM

I'm very eager to move off of pinecone, purely from a simplicity perspective.

I can work around some limitations, however there are a couple big blockers.

Metadata filtering has to support more than =, !=... E.g. <, <=, >=, > ... Even a filter for in list or not in list.
Not a blocker, but having an actual view in the dashboard would be a big help
The comment I saw above about local Wrangler support removes the other blocker

Bryan•10/11/24, 11:46 AM

I can't understand why vectorize has so many limits. I don't think I can develop a production-level project under current limits.

Compared to Pinecone, which has ZERO limits as long as I pay for what I need.

Really hope one day all limits go away.

BBryan I can't understand why vectorize has so many limits. I don't think I can develop...

Isaac McFadyen•10/11/24, 2:54 PM

The Vectorize limits aren't artificial but are a result of the internal architecture/technology backing Vectorize. I expect they'll be raised over time (and in fact they were recently) but not lifted altogether.

Isaac McFadyen•10/11/24, 2:54 PM

And nothing has zero limits: Pinecone docs say they have them too, it just scales by plan (but there's an upper bound): https://docs.pinecone.io/reference/quotas-and-limits

Pinecone Docs

Quotas and limits - Pinecone Docs

Search through billions of items for similar matches to any object, in milliseconds. It's the next generation of search, an API call away.

Any roadmap for when max dimensions will be increased? I'd love to use text-embedding-3-large but it

Similar Threads

Similar Threads

Similar Threads