Orchestrate long running workers
Background information
I have a business use case that I'm trying to figure out how to implement in workers or if workers aren't a good solution for my situation.
A user can start a long running task that consists of many sequential third party api calls. We want this long running task to be both cancellable (in case the user changes their mind mid-way), and throttle-able across all users so that we don't bring down the 3rd part API. We'd want to know when a task has officially cancelled and not just when the cancel as requested, since we would likely want to start a new task after the previous one has officially completed.
Current thoughts on implementation
We'd have to break down our long running task into multiple sequential workers for a specific task, so that one continues after the previous one has ran out of CPU time. The biggest CPU sink would be just parsing the json that we get from the 3rd party api.
Maybe a durable object (that shows which tasks should be active) + a queue that has all of the requests of long running tasks? How would we be able to implement a long-running task consisting on multiple sequential workers, prerably bundled since we're mostly waiting on API calls?
1 Reply
To implement long-running sequential tasks. Could we implement a service binding so that the previous long-running task worker can 'kick-off" the next one?