Fly.io express app (Chicago) slow queries to Neon (Ohio)
I'm trying to figure out what the latency I'm experiencing is caused by. I'm comparing Fly postgres with Neon. The increase in latency is insane, and so I doubt the regional differences is the issue here. Cold starts aren't the issue either, since I've done queries in quick succession.
NeonDB scenario:
Fly postgres scenario:
I've been able to reproduce this 10x increase in latency consistently. I've done
EXPLAIN ANALYZE and the query is not part of the issue.
The following scenario is 10 identical queries for a non-existing row (SELECT), happening over the course of 1 minute.
From Fly (Chicago) => Neon (Ohio):
From local (in sweden) => Neon (Ohio):
Something seems way off here, and I'm unsure what it is. Fly postgres seems WAY quicker, and I'm not sure if it's because it's not using a serverless model, or if it's because of the region also being ORD (Chicago), or if it's something else.
Sorry for the long post. Any assistance would be greatly appreciated.11 Replies
complex-teal•2y ago
This is very bizarre. What client are you using to interact with your database? I think I'll have to perform some network inspection to see what's going on.
fair-roseOP•2y ago
I'm just using
console.time prints from my node app using sequelize-typescript. As soon as I switched from NeonDB (feature branch) to Fly postgres in the same region, the performance was insanely improved (code being identical). If you'd like I can setup the scenario again.
This is roughly the connection string I used for the tests:
It seems more expensive to run their high availability cluster (Fly postgres). Also it's not managed, and I want backups... Neon seems great to start out. But this latency can't be ignored...
Worth noting is that write speed was also increased similarly.complex-teal•2y ago
Thanks. I'll try and test this out on Monday
Hopefully we can get you unblocked here soon
fair-roseOP•2y ago
Thanks a bunch. I almost launched on fly postgres, but I felt like this just didn’t make much sense, so here I am trying to learn what’s up 😂
Update: I can't seem to reproduce this now. I did have issues with
googleapis causing OOM issues because of it including absurd amounts of static strings which took up heap memory. And I recently fixed that, which sped up the app booting, and it also seems to have sped up the queries. A similar findOne ORM query now takes 10-20ms, which seems more reasonable.
Still not sure how it affected the query speed this much though...complex-teal•2y ago
Huh
How peculiar
Thanks for the update
I'll see if we can run a continuous test suite from fly. Seems important
fair-roseOP•2y ago
Yes! That would be great. Most notably I'm curious since the regions aren't 1:1. Meaning chicago to ohio... Not sure if that's gonna be an issue. Now it seems performant (except for the cold starts, but that's the idea with Neon to save on money I guess)
fair-roseOP•2y ago
For more information, the related issue (I think this is what fixed the slowness) is https://github.com/googleapis/google-api-nodejs-client/issues/2187
GitHub
TypeScript typings cause very high memory usage in tsc · Issue #218...
When importing googleapis within a TypeScript project, tsc takes up a large amount of memory (500 MB+). This appears to be the case because when googleapis is imported, it imports the type definiti...
fair-roseOP•2y ago
Transpiling my small express app (ts => js) went from 8 seconds to 1 second (roughly). Initialisation of the app is also immensely improved (connecting to db, truncating tables with
sequelize.sync).complex-teal•2y ago
oh, does fly offer 'native' typescript support?
fair-roseOP•2y ago
Not that I'm aware of. I transpile as part of a custom Dockerfile (just by running
tsc) which is then deployed on their machines.complex-teal•2y ago
👍