Cloudflare Developers•2y ago

Can I have more than 1 worker consumming from the same Queue?

Ddsp Can I have more than 1 worker consumming from the same Queue?

rcoopr•7/18/24, 9:33 PM

how would it choose which worker to use? what's the purpose?

Rrcoopr how would it choose which worker to use? what's the purpose?

dspOP•7/19/24, 2:21 AM

Concurrency. Can't make it work as the docs explain. My curretlly workaround is to have 10 queues with producers and consummers

Ddsp Concurrency. Can't make it work as the docs explain. My curretlly workaround is ...

Pranshu Maheshwari•7/19/24, 2:56 PM

What's the issue you're having right now?

You can setup pull consumers (https://developers.cloudflare.com/queues/configuration/pull-consumers/) if you want to manually control the level of concurrency. But the default push based consumers should automatically scale up your consumer workers based on demand.

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

PPranshu Maheshwari What's the issue you're having right now? You can setup pull consumers (https:...

dspOP•7/19/24, 8:34 PM

Hi @Pranshu Maheshwari. How are you doing?

The push based should, but it doesn't

My currently workaround is to have 10 producers and consumers and distribute between then

Here is the config:

[[queues.producers]]
binding = "WEBHOOK_QUEUE_1"
queue = "webhook-1"

[[queues.producers]]
binding = "WEBHOOK_QUEUE_2"
queue = "webhook-2"

[[queues.producers]]
binding = "WEBHOOK_QUEUE_3"
queue = "webhook-N"

[[queues.consumers]]
queue = "webhook-1"
max_batch_size = 10
max_batch_timeout = 2 
max_retries = 3
retry_delay = 30 
dead_letter_queue = "webhook-dlq"

[[queues.consumers]]
queue = "webhook-2"
max_batch_size = 10
max_batch_timeout = 2 
max_retries = 3
retry_delay = 30 
dead_letter_queue = "webhook-dlq"

[[queues.consumers]]
queue = "webhook-N"
max_batch_size = 10
max_batch_timeout = 2 
max_retries = 3
retry_delay = 30 
dead_letter_queue = "webhook-dlq"

[[queues.producers]]
binding = "WEBHOOK_QUEUE_1"
queue = "webhook-1"

[[queues.producers]]
binding = "WEBHOOK_QUEUE_2"
queue = "webhook-2"

[[queues.producers]]
binding = "WEBHOOK_QUEUE_3"
queue = "webhook-N"

[[queues.consumers]]
queue = "webhook-1"
max_batch_size = 10
max_batch_timeout = 2 
max_retries = 3
retry_delay = 30 
dead_letter_queue = "webhook-dlq"

[[queues.consumers]]
queue = "webhook-2"
max_batch_size = 10
max_batch_timeout = 2 
max_retries = 3
retry_delay = 30 
dead_letter_queue = "webhook-dlq"

[[queues.consumers]]
queue = "webhook-N"
max_batch_size = 10
max_batch_timeout = 2 
max_retries = 3
retry_delay = 30 
dead_letter_queue = "webhook-dlq"

Where N is just the following configs for both producers and consummers until 10.

If you have any tips ...

marko.333•7/20/24, 7:42 PM

Is there any plan to enhance the queue api to query the number of messages in the queue, or at least if it's empty or not? thx.

marko.333•7/21/24, 12:58 PM

There was a bit of a thread yesterday that discussed ongoing work to support dynamic provisioning/binding of D1 db's.
I was wondering if something similar is being worked on for Queue's (i'm hopeful

)?
d1-database

Mmarko.333 Is there any plan to enhance the queue api to query the number of messages in th...

John Spurlock•7/21/24, 3:01 PM

you can query the avg message size and/or avg number of messages in the backlog over time for a given queue using the graphql api (

queueBacklogAdaptiveGroups

queueBacklogAdaptiveGroups

)

https://developers.cloudflare.com/analytics/graphql-api/

WWallslide Is there a way to set a throughput limit. Like, don’t process more than 10 messa...

Shivek•7/21/24, 4:41 PM

You can set the batch size to 10 in wrangler toml?

Ddsp Each batch is processed only in sequence. I thought the concurrency feature (ma...

Shivek•7/21/24, 4:43 PM

Did you confirm this ? Are queue batches processes sequentially? I was assuming they would be processed parallely.

Ie, if the batch size is 100, and I have 1000 items, 10 workers will spawn and process all the items in one go

Ddsp Hi @Pranshu Maheshwari. How are you doing? The push based should, but it doesn...

Shivek•7/21/24, 4:46 PM

How did you verify that your consumers are not scaling?

berkinovish•7/22/24, 9:45 AM

I have a queue with concurrency = 1, because we're rate limited by an external API. I'd like to set up a system, where as soon as the call to the external API is completed, a new message gets pulled from the queue, while the previous worker is still handling persisting the data to the database.

Right now, if you

ack()

ack()

a message, while you're still handling some stuff inside a

ctx.awaitUntil()

ctx.awaitUntil()

, the queue will wait with handling the next batch until the

waitUntil

waitUntil

is fully handled.

Any idea on how we could solve this with queues?

Bberkinovish I have a queue with concurrency = 1, because we're rate limited by an external A...

Hard@Work•7/22/24, 9:48 AM

Have two queues, one that handles API calls, then another that handles responses?

HHard@Work Have two queues, one that handles API calls, then another that handles responses...

berkinovish•7/22/24, 9:52 AM

This works indeed, but pushing to the queue gets also awaited. Pushing a message on the queue can take up to 1 second for us

Bberkinovish This works indeed, but pushing to the queue gets also awaited. Pushing a message...

Hard@Work•7/22/24, 9:53 AM

Maybe set the batch sizes to be larger, so you can run multiple one after another?

Hard@Work•7/22/24, 9:53 AM

Not a perfect fix, but should help a bit

HHard@Work Not a perfect fix, but should help a bit

berkinovish•7/22/24, 9:55 AM

Yeah, would help a bit, was wondering if we could get even more out of it. Batching + pushing to a separate queue is the best we can do then probably

Bberkinovish I have a queue with concurrency = 1, because we're rate limited by an external A...

dspOP•7/22/24, 12:11 PM

That's exactally my problem. concurrency is a Myth. you have to wait for the whole batch to finish.

Ddsp That's exactally my problem. concurrency is a Myth. you have to wait for the who...

Matt Silverlock•7/22/24, 10:28 PM

That’s not true: we don’t (currently) autoscale if you have a consumer that is especially slow - the “wall clock” time of the consumer doesn’t yet feed into our autoscaling alg.

Batches are consumed concurrently.

You could consider working around this in the meantime by setting concurrency = 20 (the current maximum).

MMatt Silverlock That’s not true: we don’t (currently) _autoscale_ if you have a consumer that is...

dspOP•7/23/24, 12:24 AM

Already tried. Doesn't work

Ddsp Already tried. Doesn't work 😦

Matt Silverlock•7/23/24, 12:25 AM

Have you opened a ticket so we can take a deeper look? (can’t see if you have shared your account & queue IDs here)

MMatt Silverlock Have you opened a ticket so we can take a deeper look? (can’t see if you have sh...

dspOP•7/23/24, 12:26 AM

I think It's by design on your solution. I have some messages that take 20+ seconds to priocess and until the're not finished, another batch can't come.

Ddsp I think It's by design on your solution. I have some messages that take 20+ seco...

Matt Silverlock•7/23/24, 12:26 AM

It is not by design (I am telling you this concretely).

Matt Silverlock•7/23/24, 12:27 AM

If we’re not scaling out when a consumer is slow, it’s a bug.

MMatt Silverlock It is not by design (I am telling you this concretely).

dspOP•7/23/24, 12:28 AM

So, I did what you said on the beginning. And because it was not working, I created 10+ producers and also 10+ consummers and balance the messages between then (e.g. I did by myself manually)

Ddsp So, I did what you said on the beginning. And because it was not working, I crea...

Matt Silverlock•7/23/24, 12:29 AM

Yes, but I am telling you we don’t think you should have to do this. “By design” assumes explicit intent

Ddsp So, I did what you said on the beginning. And because it was not working, I crea...

Matt Silverlock•7/23/24, 12:29 AM

If you want us to dive deeper help us help you! queues

MMatt Silverlock Yes, but I am telling you we don’t think you should have to do this. “By design”...

dspOP•7/23/24, 12:29 AM

Oh, got it. But how can we solve this?

Brett Willis•7/24/24, 3:04 AM

Anyone notice that when using "List messages" for a queue in the dashboard, it never seems to respect the "Maximum batch size" entered there? E.g. if I enter "50" it returns 10 messages (when >50 are in the backlog), and if I enter "1" it returns 10 messages.

kev-ac•7/25/24, 8:45 PM

The 400msg/s limit of a queue is bound to the single producer, correct? If I bind for example 10 queues to the same producer/consumer worker and distribute messages among them I could handle 4000msg/s in a best-case scenario?

Kkev-ac The 400msg/s limit of a queue is bound to the single producer, correct? If I bin...

Pranshu Maheshwari•7/26/24, 4:09 PM

the limit is per queue. so you can manually shard / distribute messages across multiple queues yes, to get higher throughput. It's what a lot of folks do today. Although... higher queue limits are coming very soon!

PPranshu Maheshwari the limit is per queue. so you can manually shard / distribute messages across m...

kev-ac•7/26/24, 4:12 PM

Thanks for the clarification! What I did not notice before is that I'm also really limited by a separate factor here: The max. 6 outbound connection limit. The use case (votings) will produce multiple thousand messages in a minute or so. But the consumer side cannot push messages (in this case to an external backend) fast enough as it is limited to a maximum of 6 concurrent connections. And even worse it won't scale up the consumer concurrency from 1.

Kkev-ac Thanks for the clarification! What I did not notice before is that I'm also real...

Pranshu Maheshwari•7/26/24, 6:50 PM

ah gotcha, yeah the limit of 6 outbound connections is a hard one unfortunately.

Your consumers should scale up though; at the end of processing each batch of messages, queues checks if the consumer concurrency should be increased (based on looking at the backlog). There are a few reasons why it might not autoscale (listed on this page in the docs: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/)

Lmk in case you're having trouble with autoscaling and I can help (just send me over your account ID and queue ID, both are safe to share here)

0xthierry•7/26/24, 7:30 PM

hey guys, I'm using cf queue, but I'm experiencing some delay to delivery the message, after publishing the message in the queue, the worker starts to process it after +30s. Do you have any idea about it?

João Castro•7/27/24, 5:33 AM

Same here, sometimes i'm experiencing minutes of delay for messages to be consumed

ZZA•7/29/24, 9:44 AM

Hello guys,
i'm trying to add event notifications to one of my buckets using the command

npx wrangler r2 bucket notification create <BUCKET_NAME> --event-type <EVENT_TYPE> --queue <QUEUE_NAME>

npx wrangler r2 bucket notification create <BUCKET_NAME> --event-type <EVENT_TYPE> --queue <QUEUE_NAME>

i'm getting a bucket not found error, because my bucket's jurisdiction is EU. but i can't see to find how i can specify that to

wrangler r2 bucket notification

wrangler r2 bucket notification

any one has an idea how i can get this to possibly work ?

ZZZA Hello guys, i'm trying to add event notifications to one of my buckets using th...

kev-ac•7/29/24, 10:33 AM

Don't quote me on that but it may be possible that event notifications don't work on jurisdictional buckets yet. Super Slurper and Logpush are not supported right now, so it may also apply to Event notifications.

00xthierry hey guys, I'm using cf queue, but I'm experiencing some delay to delivery the me...

Pranshu Maheshwari•7/29/24, 3:30 PM

Could you send me your queue ID & acct ID? both are safe to share here

PPranshu Maheshwari Could you send me your queue ID & acct ID? both are safe to share here

João Castro•7/29/24, 3:41 PM

Since I seem to have a similar problem, i'm also sending my IDs in case it helps

Queue: 26149965a00b4a65a047794b2a28b3f2
Account: 978f7d8fe0d7c314743586a46e505f14

João Castro•7/29/24, 3:43 PM

This happened while running some load tests. Some messages took 20+ minutes to be processed

Kkev-ac Don't quote me on that but it may be possible that event notifications don't wor...

Pranshu Maheshwari•7/29/24, 6:20 PM

that's correct! thanks for highlighting this, I'll add it to the docs

Ben•7/30/24, 12:41 AM

So I see that we can have 10,000 queues per account enabling easy sharding:

More queues per account - up to 10,000
Developers building on Queues can now create up to 10,000 queues per account, enabling easier per-user, per-job and sharding use-cases.

Refer to Limits to learn more about Queues’ current limits.

What is the recommendationf for doing this?

Is it a worker-per-tenant situation

Or a binding-per-tenant.

Are there any resources for these approaches or are they still in progress thanks!

PPranshu Maheshwari the limit is per queue. so you can manually shard / distribute messages across m...

josh•7/30/24, 7:59 PM

is there a rough ETA when we will see the higher limits?

Jjosh is there a rough ETA when we will see the higher limits? 😆

Pranshu Maheshwari•7/31/24, 6:56 PM

we're actively working on it

i'll share news soon once I have concrete dates

rbravo•8/1/24, 3:26 AM

Can I have multiple consumers in one Worker, each one processing different queues?

Context:

I'm reading this in the docs:

Best practice
Configure a single consumer per queue. This both logically separates your queues, and ensures that errors (failures) in processing messages from one queue do not impact your other queues.

josh•8/1/24, 9:13 AM

I believe you can only have one consumer per worker but that worker can process multiple queues

“Notably, you can use the same consumer with multiple queues. The queue handler that defines your consumer Worker will be invoked by the queues it is connected to.”

lupin•8/1/24, 10:22 AM

Hey team ! I hope you're doing great ?

Is there an API to see how many data are remaining in the queue ? I only found this:

- https://developers.cloudflare.com/queues/observability/metrics/

Should we use the Get average Queue backlog over time period and query for the last 5 minutes ?

Cheers !

Cloudflare Docs

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

[[queues.producers]] binding = "WEBHOOK_QUEUE_1" queue = "webhook-1" [[queues.producers]] binding = "WEBHOOK_QUEUE_2" queue = "webhook-2" [[queues.producers]] binding = "WEBHOOK_QUEUE_3" queue = "webhook-N" [[queues.consumers]] queue = "webhook-1" max_batch_size = 10 max_batch_timeout = 2 max_retries = 3 retry_delay = 30 dead_letter_queue = "webhook-dlq" [[queues.consumers]] queue = "webhook-2" max_batch_size = 10 max_batch_timeout = 2 max_retries = 3 retry_delay = 30 dead_letter_queue = "webhook-dlq" [[queues.consumers]] queue = "webhook-N" max_batch_size = 10 max_batch_timeout = 2 max_retries = 3 retry_delay = 30 dead_letter_queue = "webhook-dlq"

Can I have more than 1 worker consumming from the same Queue?

Similar Threads

Similar Threads

Similar Threads