Workflows vs Queues
So I am really confused about the use cases of Workflows vs Queues. Seems like anything you could do in one you could also do in the other. Are there specific parameters (like limits, costs, performance, etc.) that I should keep in mind when deciding?
7 Replies
Queues trigger workers while workflows guarantee durable execution, with a workflow you can run multiple steps with different retry policies for each.
Queues can keep messages guaranteeing deliver.
There are a bunch of users that have queues that are consumed and trigger workflows from messages read.
Hope this helps
Actually that's the pattern I've been using, A Worker places an event onto a Queue, then a consuming Worker just passes that event to a Workflow that does the longer running tasks. My sense is the workflow will be more cost optimized for orchestration and longer running multi-step things, since you are only charged when it's doing stuff (I think)
Yup that sounds about right
for those consuming a Queue then triggering a Workflow, why not trigger the Workflow directly?
1. You may (and many often do) want to batch up smaller events and have a Workflow process them together
2. a queue gives you a buffer: get a burst of events that exceeds how many workflows you can run concurrently and the queue allows you to smooth it out (and/or retry)
3. Preprocess events, throw some out, validate them, etc - before kicking off a workflow
At a smaller scale - nothing wrong with just going direct to a Workflow.
that makes sense. thank you
The pattern I'm using is, I have a Worker that accepts webhooks from 3rd party systems. All it needs to do is accept the webhook, place it on a queue and return a HTTP 202. This keeps the event ingestion layer simple and focused purpose. Then I can have one or more workers starting longer running workflows processing the events. This is basically a well know/best practice async event processing pattern.