Long-running processing with SQL
Hi everyone,
I've got a feature request for a client where they want to regenerate potentially tens of thousands of SQL records on-demand within a few seconds up to a minute or two at max. In simple terms, they have a found set of Jobs where some cost items can be calculated based on the work done, materials, mileage, etc. and those items need to be stored/overwritten in SQL.
As a naive approach I tried various methods like
Promise.all() to create a bunch of Promises to be executed in parallel, but I quickly ran into a bunch of errors in Prisma, mainly due to the db running out of connections in the pool/timing out when fetching connections, etc.
I know WebDevCody has put out a few videos (like this one) where he uses SQS queues to generate thousands of PDFs in only a few seconds. I need something that functions exactly like this, but somehow work around the DB connection limitations.
I've been hacking at this for a couple weeks and would really appreciate any guidance. Thanks in advance!Web Dev Cody
YouTube
How I generated 10,000 PDFs in seconds using SST
š T3 Stack Tutorial: https://1017897100294.gumroad.com/l/jipjfm
š¤ SaaS I'm Building: https://www.icongeneratorai.com/
š¬ Discord: https://discord.gg/4kGbBaa
š Newsletter: https://newsletter.webdevcody.com/
š GitHub: https://github.com/webdevcody
šŗ Twitch: https://www.twitch.tv/webdevcody
š¤ Website: https://webdevcody.com
š¦ Twitter: https://twitt...
0 Replies