Hey Cloudflare team and community, I
Hey Cloudflare team and community, I have a large dataset of
Limits: https://developers.cloudflare.com/vectorize/platform/limits/
Changelog: https://developers.cloudflare.com/vectorize/platform/changelog/
Given my vector dimensions (256) and the new limit of 5 million vector dimensions per index, my understanding is that I could potentially store up to 19,531 vectors per index (5,000,000 / 256 = 19,531.25). Is this correct?
If so, I would need approximately 25,600 indexes to store all 500M vectors (500,000,000 / 19,531 ≈ 25,600). However, this seems impractical given the current limit of 100 indexes per account.
My questions are:
1. Is my understanding of the "5 million vector dimensions" limit correct? Or does this mean something different? I wish to insert 5M vector for each index based on Limits page statement:
2. If my understanding is correct, what would be the best approach to handle such a large dataset with Vectorize? Given the current beta limit of 5 million vectors per index (on limits page, not changelog) I am proposing to distribute your data across 100 indexes, each containing 5 million vectors. My insertion strategy involves using a modulo operation to determine which index a vector should be inserted into. For querying, I plan to search all 100 indexes in parallel and then aggregate and rank the results.
3. Are there plans to increase the number of indexes allowed per account? If so wha tis maximum?
Any advice or insights would be greatly appreciated. Thank you!
500M vectors256 dimensionsLimits: https://developers.cloudflare.com/vectorize/platform/limits/
Changelog: https://developers.cloudflare.com/vectorize/platform/changelog/
Given my vector dimensions (256) and the new limit of 5 million vector dimensions per index, my understanding is that I could potentially store up to 19,531 vectors per index (5,000,000 / 256 = 19,531.25). Is this correct?
If so, I would need approximately 25,600 indexes to store all 500M vectors (500,000,000 / 19,531 ≈ 25,600). However, this seems impractical given the current limit of 100 indexes per account.
My questions are:
1. Is my understanding of the "5 million vector dimensions" limit correct? Or does this mean something different? I wish to insert 5M vector for each index based on Limits page statement:
Maximum vectors per index= 5M2. If my understanding is correct, what would be the best approach to handle such a large dataset with Vectorize? Given the current beta limit of 5 million vectors per index (on limits page, not changelog) I am proposing to distribute your data across 100 indexes, each containing 5 million vectors. My insertion strategy involves using a modulo operation to determine which index a vector should be inserted into. For querying, I plan to search all 100 indexes in parallel and then aggregate and rank the results.
3. Are there plans to increase the number of indexes allowed per account? If so wha tis maximum?
Any advice or insights would be greatly appreciated. Thank you!
Welcome to the official Cloudflare Developers server. Here you can ask for help and stay updated with the latest news
86,529Members
Resources
Similar Threads
Was this page helpful?
