1 reply

Long-running processing with SQL

Hi everyone,

I've got a feature request for a client where they want to regenerate potentially tens of thousands of SQL records on-demand within a few seconds up to a minute or two at max. In simple terms, they have a found set of Jobs where some cost items can be calculated based on the work done, materials, mileage, etc. and those items need to be stored/overwritten in SQL.

As a naive approach I tried various methods like

Promise.all()

Promise.all()

to create a bunch of Promises to be executed in parallel, but I quickly ran into a bunch of errors in Prisma, mainly due to the db running out of connections in the pool/timing out when fetching connections, etc.

I know WebDevCody has put out a few videos (like this one) where he uses SQS queues to generate thousands of PDFs in only a few seconds. I need something that functions exactly like this, but somehow work around the DB connection limitations.

I've been hacking at this for a couple weeks and would really appreciate any guidance. Thanks in advance!

YouTubeWeb Dev Cody

How I generated 10,000 PDFs in seconds using SST