Cloudflare Developers•7mo ago

Hey team ! I'm trying to edit the

Hey team ! I'm trying to edit the concurrency on some of my queues, but the option doesn't exist anymore:

lupinOP•6/8/25, 6:35 AM

Moreover, the queue concurrency was at 250, but now it max at 20

1984 Ford Laser•6/8/25, 10:54 PM

You set concurrency when adding a Consumer to the Worker

Sid•6/9/25, 10:45 PM

The actual concurrency value goes up over time based on how many unacknowledged messages you have and if your queue() handler isn’t throwing errors. Maybe after 20 you just ran out of messages?

lupinOP•6/10/25, 9:01 AM

@Sid , unlikely since we had 100k messages in the queue and it was stalling. Sorry I forgot to add this detail in the thread

lupinOP•6/10/25, 9:02 AM

I'm currently testing several configurations, but seems like that from time to time the queue is randomely hanging and will not try to add more consumers. The error rates is the same so I don't think it comes from that ?

lupinOP•6/10/25, 9:05 AM

The new queue I'm using has the following configuration:

batch_size: 50,
max_concurrency: 250,
max_retries: 3,
max_wait_time_ms: 0

batch_size: 50,
max_concurrency: 250,
max_retries: 3,
max_wait_time_ms: 0

The message retention is the default one of 345600 seconds345600 seconds. I'll update the results in the thread

lupinOP•6/10/25, 9:14 AM

With this setup the queue doesn't scale that much ?

The behaviour we had before the problem, was that the queue would scale to 100 concurrency and with the same amount of messages will clean it without any issues. I'm doing all my tests with the same exact messages and the same Workers behind it. There is no notable difference

lupinOP•6/10/25, 9:18 AM

@1984 Ford Laser thanks btw for the feedback ! I thought it wasn't on the UI anymore but indeed you can change it changing the attached consumer ! Nice catch :cooldoge:

Sid•6/10/25, 10:05 AM

Can you give me a queue ID?

lupinOP•6/10/25, 10:08 AM

f024d80269db4736a73e60db222dec7d

lupinOP•6/10/25, 10:08 AM

I changed the batching to 1 instead of 50 and the concurrency went up to 130

lupinOP•6/10/25, 10:10 AM

Thanks for looking into this @Sid

Sid•6/11/25, 6:51 PM

Hey sorry for the delay. I looked into this and things do seem to be working as expected.
I will say though that there are a bunch of improvements we can make here. I’ve been doing some of that on the producer side recently, and I can probably get some quick wins on the consumer side as well.

lupinOP•6/12/25, 8:22 AM

I think the problem was the batching @Sid . When going from 50 to just 1, the queue managed to empty normally. This is a strange behaviour because for 1 year we had a batching of 100 with 20 concurrency and it worked like a charm. Maybe something changed on that part ?

Sid•6/12/25, 10:12 AM

Hm, we haven’t changed anything on that front yet.
If you’re saying that lowering your batch size increased your concurrency, then I would expect that. Consumer concurrency calculation isn’t straightforward right now so it’s hard to explain here, but if you really do value concurrency that much then lowering the batch size is the right thing to do.

lupinOP•6/15/25, 1:18 PM

@Sid | Queues, R2 it's pretty random on our side still. The 1 batch fixed for a few hours, but when doing it again today it takes 2 hours to do 70k messages, where we had 100k in 30 minutes before (we didn't change the code from the infra). I've reverted now to the 100 batch messages and see if it changes anything, but seems like the queues are not scaling like before

lupinOP•6/15/25, 4:12 PM

Hey @Sid | Queues, R2, I'm making a lot of different tests, but it seems like every time the queues are hanging. The consumer has a minimal code where it just do a 3 http request that shouldn't take too much time. Also, the queue were doing 250 concurrency on a 100 batch. Now the concurrency is dropping randamely, the backlog is not emptying as fast as before, and changing the parameters, or spawning a new queue is not helping at all. Here are some screenshots:

lupinOP•6/15/25, 4:12 PM

Queue Id: fa75809b889646e1bb7e87bd33fe735dfa75809b889646e1bb7e87bd33fe735d (we created a new one in case it solved something haha)

Sid•6/15/25, 4:24 PM

Can I get back to you on Monday? I’ll check how long your queue() handler is actually taking

Sid•6/15/25, 4:24 PM

3 HTTP requests seem fine. Our consumer concurrency logic (which can be improved for sure), currently looks at the rate of success of your handlers, and bumps up concurrency every 5 seconds or so

Sid•6/15/25, 4:27 PM

I will say though, we should look at the lag time instead of focusing on concurrency in general. Are you expecting the 5K seconds we predict to go much lower?

Sid•6/15/25, 4:29 PM

The number of messages (140k) also does not seem like a big deal to me. I’m unfortunately away from my laptop right now, but I’ll get back to you.

Aside: on our end we should probably look at the backlog size as well when scaling consumers up, this is something I want to do when I get some time. That should also help you predict concurrency a lot better (plus it’ll probably match your expectations better)

lupinOP•6/15/25, 6:49 PM

Yes of course @Sid ! Thanks for looking into it :D

lupinOP•6/15/25, 6:50 PM

Are you expecting the 5K seconds we predict to go much lower?

Yes definetely. It was around 700 seconds before. I don't know what happened with the queues recently ^^'

SSid The number of messages (140k) also does not seem like a big deal to me. I’m unfo...

lupinOP•6/15/25, 6:52 PM

I just always assumed that the concurrency was based on the backlog + average lag time

lupinOP•6/15/25, 6:52 PM

For additional information, here is the state of the same queue since the previous screenshot

lupinOP•6/15/25, 6:53 PM

It's not the first time we noticed that kind of weird curves (we've opened tickets in the past) but seems like the problem is still not resolved ^^'

lupinOP•6/16/25, 5:05 AM

The last 12 hours stats look like this. Before we were doing 100k messages in 30 minutes, now it's 200k messages in 6 hours

lupinOP•6/17/25, 11:40 AM

@Sid I confirm with you that unfortunately we are still having the same lag in all of our queues. Should I open a formal ticket since it's impacting our PRD and STG at this point ?

Sid•6/17/25, 12:59 PM

Yeah let’s open a ticket. It’s going to come back to me, but I can look into this. Sorry it’s been a really busy few days

lupinOP•7/1/25, 9:35 AM

Hi @Sid ! I took some time, but I finally opened the case since the problem is coming back again:

CaseId: 01630994

Thank you for your time and I'm looking forward to sort this out ! :D

lupinOP•7/11/25, 2:15 PM

Hi @Sid ! I haven't heard from the ticket and it seems that the performances are still unstable:

lupinOP•7/11/25, 2:15 PM

The queue randomly dropped to 1 concurrency even though there was still messages in the queue

Sid•7/12/25, 2:54 PM

Is this still your Queue ID? fa75809b889646e1bb7e87bd33fe735dfa75809b889646e1bb7e87bd33fe735d? I don't see any traffic on this one for the last 7 days

lupinOP•7/13/25, 12:34 PM

This problem is happening on all of our queues to be fair @Sid . I need to find back the queue Id for this speicifc one

Sid•7/13/25, 12:46 PM

Can you use the /link command to link your account here? Then I can look at what queues you have

Sid•7/13/25, 12:47 PM

Generally I’m looking for one with some traffic / backlog

Sid•7/13/25, 12:48 PM

Consumer concurrency going down might be a result of what happens in your queue() handler. That’s what I wanted to check

lupinOP•7/14/25, 9:34 PM

Thanks @Sid for pointing that out ! I wasn't aware that we could link our Cloudflare accounts. Just did it :)

lupinOP•8/3/25, 12:59 PM

Hi @Sid ! We are still experimenting the problem and I would say it's even worse than before f024d80269db4736a73e60db222dec7df024d80269db4736a73e60db222dec7d

lupinOP•8/3/25, 12:59 PM

On the span of 12 hours the queue didn't move ^^'

Sid•8/3/25, 12:59 PM

Oh man I’m so sorry I’ve been absolutely swamped, let me catch back up on your case

lupinOP•8/3/25, 1:00 PM

No worries at all I totally get that. It's just that it's production breaking at this point and we are really out of ideas ^^'

Sid•8/3/25, 1:00 PM

Your issue was that you’re not seeing as much concurrency as you’d want, right?

lupinOP•8/3/25, 1:01 PM

As you see the lag time is growing, the queue is not shrinking and the concurrency is around 10/15 instead of the 250 that it should scale to

lupinOP•8/3/25, 1:02 PM

And no matter how many messages are in the queue (20k or 2M) it will be bloated no matter what

Sid•8/3/25, 1:02 PM

So the first thing I see is that your queue consumer takes a while, a few minutes

Sid•8/3/25, 1:02 PM

IIRC you’ve already tried reducing your batch size, right?

lupinOP•8/3/25, 1:03 PM

Yes we've experimented with 1, 5, 50, 100

lupinOP•8/3/25, 1:03 PM

The consumer takes time because of the underlying HTTP requests it does, but it has no CPU time

Hey team ! I'm trying to edit the

Similar Threads

Similar Threads

Similar Threads