How do you approach long async types of operations on t3/next/vercel?
Say I need to run a process that takes 5 minutes, it gets data from DB, does some calculations, generates a CSV and drops it in s3.
4 Replies
There are a couple of ways I know of, one is with Upstash https://docs.upstash.com/redis/tutorials/job_processing
Job Processing and Event Queue with Serverless Redis | Upstash: Doc...
This tutorial shows how to use Upstash Redis for job/task processing.
The other is with trigger dev it is in beta / alpha now but you can apply and get access https://trigger.dev/
Trigger.dev - The Background Jobs framework for Next.js
Create long-running Jobs directly in your codebase with features like API integrations, webhooks, scheduling and delays.
Otherwise you can make or use an external long running service and use a webhook between the service and next js when the job finishes
A relatively inconvenient option:
I run a separate node/go service and expose it as a private API.
This just entirely decouples the delegation of long running jobs from the primary front-end stack.