Hi there! I've been porting some python code I had using inngest, and some other built with queues,
Hi there! I've been porting some python code I had using inngest, and some other built with queues, and I have a bunch of feedback that might be useful.
For reference, the code that I've migrated to workflows usually does something like this:
Pao
For reference, the code that I've migrated to workflows usually does something like this:
- fetches a bunch of resources from the DB
- (fanout) for each resource it does an http call to an API and then it saves some data to the DB
- (fanin) once everything is done, it makes more changes to the DB
- The 256 step limit means that I can't build this as a single workflow. I have to add a queue to implement the "fanout". The alternative is to do a
Promise.allSettled, but then you don't get to retry failed workflow creation. - The http call has a rate limit, and unlike queues, there is no way to specify concurrency/batch size. Since I moved the "fanout" to a queue, this is where I implement the rate limit, but it would be great to be able to describe the whole workflow without queues.
- Because of moving the fanout to queues, there is no way to do the fanin part since it can't be known when all the queues have been processed.
- There is no way to compose workflows. This means that if you have a piece of workflow that is used in several places, you have to abstract away the steps. It would be great if workflows could return values so a workflow could call another workflow like a step.
Pao
