The v2 docs indicate that querying indexed metadata can truncate long text values. I understand not

The v2 docs indicate that querying indexed metadata can truncate long text values. I understand not wanting to define the exact value in the API, although is there a guideline for this truncation threshold? I would like to store the original text associated with the vector. In my use case, these text values are generally a few hundred words or less.

CC J The v2 docs indicate that querying indexed metadata can truncate long text value...

Jerome•9/3/24, 2:42 PM

You can store up to 10KiB of metadata per vector; vectorize will either accept the data if it fits, or reject the upsert if not. The truncation happens in the metadata index at 64B, meaning you can filter indexed metadata properties on their first 64B. The original vector metadata is never truncated, and can be obtained verbatim on vector query by specifying the "metadata": "all""metadata": "all" option.

JJerome You can store up to 10KiB of metadata per vector; vectorize will either accept t...

C JOP•9/3/24, 5:21 PM

Thank you for those details. That’s really helpful. So basically ~64 ASCII characters before truncation when using indexed metadata?

The challenge with metadata: all option for me is that it limits to topK 20 results. I’m hoping to return closer to 40 results, as I do a rerank and then trim the set before presenting to the LLM. Metadata: indexed would let me return topK 100 but the truncation cutoff makes it less useful.

Is the topK max of 20 anticipated to improve in the future with metadata: all? Otherwise I will need to completely re-architect and store metadata independently of vectors.

maro•9/3/24, 8:38 PM

Hey guys! I'm aware that Vectorize is currently in beta, but I'm curious about its reliability for production use. My specific use case involves a chatbot that serves approximately 1 million users per month. If Vectorize isn't suitable, could you recommend any other alternatives? Pinecone? pgvector?

scotto•9/4/24, 1:52 AM

Any plans to support arrays in metadata filtering ?

hannojg•9/5/24, 8:46 PM

Hey, quick question: its expected that on the web dashboard you don't see any of the vector indexes you've created (while in wrangler you do) ?

hannojg•9/5/24, 8:47 PM

ah that has already been answered here How can I bind it to a project without thanks!

hannojg•9/5/24, 8:49 PM

~~Is there a limit to how many vector indexes we can create?~~
Edit: found the limits docs, all good

Mmaro Hey guys! I'm aware that Vectorize is currently in beta, but I'm curious about i...

yevgen•9/6/24, 12:49 PM

@maro Vectorize is ready for production use, we are planning to transition out of beta soon.

Yyevgen @maro Vectorize is ready for production use, we are planning to transition out o...

hannojg•9/6/24, 12:57 PM

Cool! Are there any early details on how the limits will increase?

ac•9/6/24, 1:09 PM

Is there any way to upgrade a v1 Vectorize index to v2, or do I need to create a completely new one?

Original message was deleted

Isaac McFadyen•9/6/24, 9:41 PM

TIL that you can link a slash command in a message

Hhannojg Cool! Are there any early details on how the limits will increase? 😊

yevgen•9/7/24, 9:38 AM

So far we haven't observed anyone using Vectorize to encounter our current limits, what exactly you have in mind?

Yyevgen So far we haven't observed anyone using Vectorize to encounter our current limit...

cloudkite•9/7/24, 8:57 PM

Would love to be able to dynamically create & bind an index per customer/tenant. Is that something that’s on the roadmap?

Yyevgen So far we haven't observed anyone using Vectorize to encounter our current limit...

hannojg•9/8/24, 1:24 AM

In our use case we don’t have a lot of data belonging to one category, but a lot of smaller data belonging to many segments.
So I am wondering how I can segment the vectors. The options I see are:
- creating an index per segment
- using namespaces in the same index
- metadata filtering (less preferred)

Both indexes and namespaces have quite a tight limit for our use case

Ccloudkite Would love to be able to dynamically create & bind an index per customer/tenant....

hannojg•9/8/24, 1:27 AM

Kind of like this - if we already have 20.000 customers, and each customer has ~15 projects for which I’d need separate segments, it’s hard to do that with the current limits

yevgen•9/9/24, 8:45 AM

I think metadata filtering is the best way forward. If you prefer using namespace for whaterever reason (nothing special other then we index it by default) we can lift # of namespaces limit for your account.

yevgen•9/9/24, 8:48 AM

Creating an index per segment could be an option as long as your index size will be at least 50k vectors, otherwise one index for all data is probably a better match.

Ccloudkite Would love to be able to dynamically create & bind an index per customer/tenant....

yevgen•9/9/24, 8:53 AM

yes, Q1 2025 we plan to ship support for Vectorize dynamic bindings.

cosbgn•9/9/24, 1:26 PM

Can someone give me a brief guide/bullet points on how to use vectorize for a saas?

- Should I create a namespace per user and query that namespace?
- Should I just add them all in one namespace and query for metatags?

What's the best proactice?

cosbgn•9/9/24, 1:28 PM

1000 namespaces for now it would be enough to have one per user, but that can eventually be increased, right?

cosbgn•9/9/24, 1:29 PM

Also, is there any restriction on saving the original text in the metadata? It would save me an extra call to D1

cosbgn•9/9/24, 1:30 PM

10KiB of metadata is enough to store most text

cosbgn•9/9/24, 1:31 PM

all of it actually + some ids and some other small info

Ccosbgn 1000 namespaces for now it would be enough to have one per user, but that can ev...

yevgen•9/9/24, 5:32 PM

yes, it could be increased.

ac•9/11/24, 2:03 PM

Hi, when trying to get vector values the Vectorize API docs say

values

values

is "an array of number, Float32Array, or Float64Array as the vector embedding itself."

Can someone tell me what actually determines which one of these types the vector is?

Aac Hi, when trying to get vector values the Vectorize API docs say `values` is "an ...

Jerome•9/11/24, 5:57 PM

Hi! I believe you're refering to https://developers.cloudflare.com/vectorize/best-practices/query-vectors/; the format you use to query your DB through a binding can be either of these, it'll be interpreted the same way, as a floating point number array. You also don't have to use the type that matches the one you used when inserting/upserting, all number formats are interoperable for insert/upsert and query.

JJerome Hi! I believe you're refering to https://developers.cloudflare.com/vectorize/bes...

ac•9/11/24, 8:31 PM

Thanks, but I'm talking about the return value. It seems like there are instances when it is Float32Array or Float64Array instead of number[], but I'm trying to determine how to know which it will be.

ac•9/13/24, 8:53 PM

I am getting 504 Gateway Timeouts intermittently when running wrangler vectorize listwrangler vectorize list . Is there an ongoing incident?

ac•9/13/24, 8:55 PM

Also getting this intermittently on deploys, which has never happened before:

Error: Failed to publish your Function. Got error: binding <vectorize index name> of type vectorize failed to generate: internal error; please try again later or contact support

Error: Failed to publish your Function. Got error: binding <vectorize index name> of type vectorize failed to generate: internal error; please try again later or contact support

Aac I am getting 504 Gateway Timeouts intermittently when running `wrangler vectoriz...

yevgen•9/15/24, 12:42 PM

We saw intermittent issues with one of the service instances running in our Core datacenter. It has resolved now.

cosbgn•9/15/24, 4:07 PM

Any way to filter metadata as array includes?

In my use case I have some files and they have tags like ["books", "work"], etc these tags are set by the users and I don't know them in advance.

Ideally I would be able to query embeddings only for "work" or only for "books" ]

cosbgn•9/16/24, 8:51 AM

Any way to delete a namespace and all the vectors inside it via rest API? doesn't seem to be documented

charl.dev•9/16/24, 8:32 PM

Is setting a ttl on a vector entry, similar to kv on the cards?

Have items I want to clear out like 3 months after creation. Can do like a nightly purge but would be a great feature IMO

Ccosbgn Any way to delete a namespace and all the vectors inside it via rest API? doesn'...

Sunny•9/17/24, 5:04 PM

I've been deleting through "wrangler vectorize delete <index_name>"

Sunny•9/17/24, 5:04 PM

though I'm running into an issue when trying to create a new index under the same name

Sunny•9/17/24, 5:05 PM

it gets deleted off my dashboard but I get this error when creating a duplicate index. Not sure how to fully remove it and its history

Sunny•9/17/24, 5:06 PM

🚧 Creating index: 'database-name'

✘ [ERROR] A request to the Cloudflare API (/accounts/b7b5a2f84f22862a502266ecc6abb2a0/vectorize/v2/indexes) failed.

  vectorize.index.deleted - Index name "database-name" [code: 3005]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

🚧 Creating index: 'database-name'

✘ [ERROR] A request to the Cloudflare API (/accounts/b7b5a2f84f22862a502266ecc6abb2a0/vectorize/v2/indexes) failed.

  vectorize.index.deleted - Index name "database-name" [code: 3005]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

chronark•9/18/24, 8:47 AM

is there a way to clear data in an index in the UI?
I can't see any and the docs aren't helping either

Sunny•9/18/24, 3:28 PM

I think you can only delete elements through the wrangler cli, and it’s only element by element, reference by id

Sunny•9/18/24, 3:28 PM

Otherwise, again through cli, you delete the whole index and make a new one

matt•9/19/24, 7:15 PM

Any roadmap for when max dimensions will be increased? I'd love to use text-embedding-3-large but it's 3072 dimensions.

Mmatt Any roadmap for when max dimensions will be increased? I'd love to use text-embe...

thipperz•9/20/24, 12:13 AM

you know that in the meantime you can specify 1536 dimensions in the open ai api call right?

matt•9/20/24, 3:25 AM

Yes I’m aware thank you. I’d like to take full advantage of it though.

mikibo•9/20/24, 5:51 PM

Hello there! Looking for how to use logical OR in vectorize metadata filtering. Is it possible? I've seen ANDAND (implicit) but can't find OROR.

charl.dev•9/20/24, 7:30 PM

Is vectorize available in DOs? I’ve bound it via toml and via env type but when I log this.env, I can see all my other bindings but not vectorize.

When I run wrangler dev, it does say the worker has access to vectorize

Ccharl.dev Is vectorize available in DOs? I’ve bound it via toml and via env type but when ...

charl.dev•9/20/24, 7:32 PM

Nvm was running remote flag with npm run dev lol

Eric•9/25/24, 4:24 AM

Hi all, when I delete an index, and try to create it with the same name through the API, I get

{
    "result": null,
    "success": false,
    "errors": [{
        "code": 3005,
        "message": "vectorize.index.deleted - Index name \"your_index_name\""
    }],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [{
        "code": 3005,
        "message": "vectorize.index.deleted - Index name \"your_index_name\""
    }],
    "messages": []
}

Is that by design? Is there a timelimit that it'll be purged on the backend?

The v2 docs indicate that querying indexed metadata can truncate long text values. I understand not

Similar Threads

Similar Threads

Similar Threads