guys im using the pgvector extension and
guys im using the pgvector extension and am about to onboard a large client with 10.000s of files. So far I've only had to deal with 100s of files and 10.000s of chunks. Will I know have to worry about partitioning etc or does neon.tech do some magic for me handling this number of files/chunks? And if not timescale seems to claim some major scalability benefits. Would they be a better vendor?
6 Replies
absent-sapphire•16mo ago
Do you mind sharing more details about how your database schema is structured?
@martin-full-stack Happy to ping someone from the Neon Storage team. But pretty sure they'll ask for more details when it comes to the amount of data you're working with
unwilling-turquoise•16mo ago
Hi Mahmoud, thanks! Could you elaborate on what you mean by how the scheme is structured. I'm using llama_index and integrate with the pgvectorstore if that helps
Also, quick question. Is the London region on the immediate roadmap at all?
harsh-harlequin•16mo ago
Hey Martin,
What’s the vector size? Are you using OpenAI embeddings?
If you’re anticipating 1M vectors, then Neon can handle that fairly easily.
Why were you thinking of partitioning?
unwilling-turquoise•16mo ago
Yes, using OpenAI embeddings, looking at the 10M range for number of vectors.
I'm just trying to prepare for very large datasets, not sure if partitioning is the answer or even needed tbf
As a follow up, any thought on adding london as a region?
harsh-harlequin•16mo ago
Thanks for clarifying.
We just added support for pgvector 0.7.0, and you should have no issues with 10M vectors with an hnsw index. You might want to test with a higher ef_construction. Default is 64, but 128 might yield better recall.
Happy to help there.
As for London region. We have no plans at the moment. Is it a requirement for your app?
unwilling-turquoise•16mo ago
Amazing, thanks. Sorry havent opened Discord in a while.
Well, having my server and users in London it makes sense for the db to be here as well. It's adding signifcant delay to the inference pipeline which is tough. It mentioned in the website you guys are looking into new regions and asking for feedback on whcih ones to add.
I've had some comments from users on inference speed already if it becomes a deal breaker I'll either have to move the server to Frankfurt (not ideal) or move off neon.tech entirely