Queues

Multiple queues?

Sam•11/11/24, 3:05 PM

Just one queue!

nclevengerOP•11/11/24, 3:05 PM

but you can add multiple queues to increase concurrency ... that's what we've done ...

Sam•11/11/24, 3:09 PM

Multiple queue for the same task? Is there not a better way than that?

nclevengerOP•11/11/24, 4:09 PM

it's not ideal, but it's vastly better now than it used to be before they raised all of the limits recently (it previously took 10 queues to get the currently levels) ... you could shard on say queue1queue1, queue2queue2, etc and the same worker can be consumer to all 10 or whatever number of those queues

Sam•11/11/24, 6:42 PM

How do you manage know whether a queue is "at capacity" or do you just randomly distribute?

Sam•11/11/24, 6:43 PM

assume this will scale 250 consumer workers per queue? so 10 queues = up to 2500 workers

Sam•11/11/24, 6:45 PM

Did you also try using a cloudflare worker as a pull consumer instead of the default push consumer approach?

nclevengerOP•11/11/24, 7:33 PM

for pull consumers to work you'd have to manage your own invocations/scaling, as well as you might hit the Cloudflare API limit ... as for capacity, we just randomized the queues we were sending to, and that has worked fine for us ...

Queues

Similar Threads

Similar Threads

Similar Threads