For the moment, you're going to find (
For the moment, you're going to find (which you may have already spotted) that you're going to dodge one bullet, only to be hit by the next in terms of limits.
In some ways, this has actually been a good process, because it made me really think about the data in a much deeper way than otherwise - in order to get some scale out of it, I've got a rotator system in, which binds N indexes to a worker, and I have N workers doing that. With 100 indexes and 200K vectors/index limits, that's about the only way to really push it at the moment - I would suspect/hope we are due for an update soon including limit increases.
In some ways, this has actually been a good process, because it made me really think about the data in a much deeper way than otherwise - in order to get some scale out of it, I've got a rotator system in, which binds N indexes to a worker, and I have N workers doing that. With 100 indexes and 200K vectors/index limits, that's about the only way to really push it at the moment - I would suspect/hope we are due for an update soon including limit increases.
