For the moment, you're going to find (

For the moment, you're going to find (which you may have already spotted) that you're going to dodge one bullet, only to be hit by the next in terms of limits. In some ways, this has actually been a good process, because it made me really think about the data in a much deeper way than otherwise - in order to get some scale out of it, I've got a rotator system in, which binds N indexes to a worker, and I have N workers doing that. With 100 indexes and 200K vectors/index limits, that's about the only way to really push it at the moment - I would suspect/hope we are due for an update soon including limit increases.
4 Replies
Unknown User
Unknown User10mo ago
Message Not Public
Sign In & Join Server To View
oldmanmeta
oldmanmeta10mo ago
Thanks for the update and I certainly appreciate the challenges. We're going to run with our current implementation in a version editor fashion, fully expecting breaking changes. The only thing we don't want is for release to kill of existing indexes as we would be on the hook for our customers training costs. Good luck team.👍 Apologies for appalling grammar- couldn't find my glasses 🤣
Jerome
Jerome10mo ago
We don't want is for release to kill of existing indexes as we would be on the hook for our customers training costs.
If possible and this is a concern, you should store your vectors/meta on mass storage as well btw
oldmanmeta
oldmanmeta9mo ago
Thanks for the headsup here. It's something that I will implement shortly for sure. Just another thing in a giant list of things to worry about 😉
Want results from more Discord servers?
Add your server