Hey Cloudflare team and community, I
Hey Cloudflare team and community, I have a large dataset of
Limits: https://developers.cloudflare.com/vectorize/platform/limits/
Changelog: https://developers.cloudflare.com/vectorize/platform/changelog/
Given my vector dimensions (256) and the new limit of 5 million vector dimensions per index, my understanding is that I could potentially store up to 19,531 vectors per index (5,000,000 / 256 = 19,531.25). Is this correct?
If so, I would need approximately 25,600 indexes to store all 500M vectors (500,000,000 / 19,531 ≈ 25,600). However, this seems impractical given the current limit of 100 indexes per account.
My questions are:
1. Is my understanding of the "5 million vector dimensions" limit correct? Or does this mean something different? I wish to insert 5M vector for each index based on Limits page statement:
2. If my understanding is correct, what would be the best approach to handle such a large dataset with Vectorize? Given the current beta limit of 5 million vectors per index (on limits page, not changelog) I am proposing to distribute your data across 100 indexes, each containing 5 million vectors. My insertion strategy involves using a modulo operation to determine which index a vector should be inserted into. For querying, I plan to search all 100 indexes in parallel and then aggregate and rank the results.
3. Are there plans to increase the number of indexes allowed per account? If so wha tis maximum?
Any advice or insights would be greatly appreciated. Thank you!
500M vectors, each with 256 dimensions. I've recently seen the changelog about Vectorize v2 being in public beta, which mentions support for up to 5 million vector dimensions per index but there is not vector dimension limit mentioned on limits page? I'd like some clarification and advice on how to best use this with my dataset.Limits: https://developers.cloudflare.com/vectorize/platform/limits/
Changelog: https://developers.cloudflare.com/vectorize/platform/changelog/
Given my vector dimensions (256) and the new limit of 5 million vector dimensions per index, my understanding is that I could potentially store up to 19,531 vectors per index (5,000,000 / 256 = 19,531.25). Is this correct?
If so, I would need approximately 25,600 indexes to store all 500M vectors (500,000,000 / 19,531 ≈ 25,600). However, this seems impractical given the current limit of 100 indexes per account.
My questions are:
1. Is my understanding of the "5 million vector dimensions" limit correct? Or does this mean something different? I wish to insert 5M vector for each index based on Limits page statement:
Maximum vectors per index= 5M2. If my understanding is correct, what would be the best approach to handle such a large dataset with Vectorize? Given the current beta limit of 5 million vectors per index (on limits page, not changelog) I am proposing to distribute your data across 100 indexes, each containing 5 million vectors. My insertion strategy involves using a modulo operation to determine which index a vector should be inserted into. For querying, I plan to search all 100 indexes in parallel and then aggregate and rank the results.
3. Are there plans to increase the number of indexes allowed per account? If so wha tis maximum?
Any advice or insights would be greatly appreciated. Thank you!
