Diagnosing PGVector query variance
I'm running into a huge variance in response times (250ms -8000ms with an average case latency is fine at ~400ms.) in a Postgres query and trying to understand the source of the issue.
We have a Neon Postgres instance with a few tables that use
pgvector for embeddings and OpenAI text-embedding-3-small embeddings. We're hosted on Vercel and using Neon through their integration.
We use this for is a query that's a hybrid embedding and popularity search. The table isn't very big, it's about 35k records. The order by can't use the HSNW index because we're doing some math with it. Basically cosine distance + custom popularity score so we're in for some trouble.
That said, it doesn't even seem like it's a case of certain inputs being slower than others in doing the calculation. Anecdotally, we see single queries being really slow, then redoing that query again being fast. Right now our cache ratio is 98.22%.
Do you have any suggestions for next places to troubleshoot?0 Replies