2 replies

Queue "consumer concurrency" feature

Queue "consumer concurrency" feature does not seem to be suitable for scenarios that rely on sequential execution.

micoozOP•9/8/24, 5:23 PM

For example,

I have some subtasks that need to be executed in sequence, each task wraped into a queue message and will take a long time to finish:

1,2,3,4,5,6

max_batch_size = 2
max_concurrency = 2

Queue will create two consumer workers to handle those messages concurrently:

1,2 -> worker-1
3,4 -> worker-2
5,6 pending

so 3,4 may be excuted earlier than 1,2:

3,4,1,2,5,6

this breaks the original sequence.

micoozOP•9/8/24, 5:39 PM

The docs: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/

Cloudflare Docs

Consumer concurrency | Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.

Pranshu Maheshwari•9/9/24, 9:56 PM

Queues today doesn't support strict ordering. But also, as Leo mentioned, concurrency doesn't really work with strict ordering anyway

Queue "consumer concurrency" feature

Similar Threads

Similar Threads

Similar Threads