Long-running worker performance
I've implemented a background sync worker, triggered by a queue. It calls an external, paginated API (dozens of sub-requests) and performs 100s of SQL queries using the socket API.
This works wonderfully, but it's very slow. Where a local execution might take 2-3 seconds, in production it takes over 60. Some syncs are ~5 minutes.
Any tips on how to diagnose the slowdown in production, or where to look to optimize? Off the top of my head, the source could be:
This works wonderfully, but it's very slow. Where a local execution might take 2-3 seconds, in production it takes over 60. Some syncs are ~5 minutes.
Any tips on how to diagnose the slowdown in production, or where to look to optimize? Off the top of my head, the source could be:
- Roundtrips to/from the DB server (which is local in dev, and supabase.co in prod)
- Worker CPU speed (don't think I'm doing anything too CPU intensive, but there is some list processing)
- Sub-request (or socket) latencies
