There's a few reasons why consumers might not autoscale (documented here: https://developers.cloudfl

There's a few reasons why consumers might not autoscale (documented here: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/#why-are-my-consumers-not-autoscaling) (cc @ajgeiss0702)

If you're seeing timeouts @ac, it most likely means your Queue consumer is taking too long to process a message. You'll need to refactor your consumer to process the messages in under 30s of CPU time

Cloudflare Docs

Consumer concurrency | Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.

PPranshu Maheshwari There's a few reasons why consumers might not autoscale (documented here: https:...

ac•11/27/24, 3:39 PM

Thanks, I will take a look. Ultimately I believe the issue is browser rendering limits/the CF Browser Renderer holding onto processes for way longer than it's supposed to, which causes the whole queue to be backed up

PPranshu Maheshwari There's a few reasons why consumers might not autoscale (documented here: https:...

ajgeiss0702•11/27/24, 6:25 PM

I saw this, and I don't think it applies:
1. Max concurrency is not set
2. No errors are reported in the queues dashboard
3. batches are processed every 15 minutes (because of the other issue I mentioned) and it hasn't really scaled up

It has scaled up to 2 concurrency a few times, but hasn't gone past that, and it didnt stay on 2 for very long

xiechao•11/30/24, 6:55 AM

I would like to ask about billing related issues. I am currently facing a problem

xiechao•11/30/24, 6:56 AM

Remaining time on 11 × Images Stream Bundle Basic Stream storage per thousand minutes

xiechao•11/30/24, 6:56 AM

What is the fee?

1984 Ford Laser•11/30/24, 8:14 AM

?crossposting

SuperHelpflare•11/30/24, 8:14 AM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Poyo•12/1/24, 1:10 PM

I'm trying to implement a simple queue. My worker sends a queue message to the consumer. I can see the message in the queue via the dashboard, but for some reason, the messages are just sitting in the queue, and processing doesn't start for 20 minutes. Do you have any ideas about what can cause this? I have logging and I can see that it is just sitting in the queue - no errors or anything.

Poyo•12/1/24, 1:12 PM

The timeouts and settings are default. I've spent a couple of hours but can't find a solution. I hope it is not an unannounced service disruption.

Poyo•12/1/24, 1:23 PM

Now it took a message to start processing - 4m 38s

Poyo•12/1/24, 6:18 PM

Okay, I found a problem - the queue exceeded the CPU limit when the puppeteer was imported.

Poyo•12/1/24, 10:37 PM

Found a way to fix it - don't use the browser inside queue but move it to a separate worker. Check the docs for Rendering Worker with Durable Object

ajgeiss0702•12/2/24, 12:45 AM

i wonder if thats the fix to my problem

ajgeiss0702•12/2/24, 3:57 AM

wow i think that might have solved my problem too

ajgeiss0702•12/2/24, 3:57 AM

so it seems like browser rendering has some issue with queues for some reason

ajgeiss0702•12/2/24, 3:58 AM

would have been nice to know this before black friday week, but better late than never

ajgeiss0702•12/2/24, 4:04 AM

its the first time ive seen this number in weeks! so beautiful!

PPoyo Found a way to fix it - don't use the browser inside queue but move it to a sepa...

ajgeiss0702•12/2/24, 4:11 AM

thank you very much. You just solved an issue i've been having for weeks and could not find a solution for!

NoGameNoFun•12/2/24, 6:54 AM

hi, i trying to set the cpu ms limit for the worker that run as consumer of a queueu, based on the documentation, it suppose can set a max limit of 15 minutes of cpu run limit, but when i try to set via wrangler or even using the dashboard, both i got is max 30,000 ms only

Aajgeiss0702 thank you very much. You just solved an issue i've been having for weeks and cou...

Poyo•12/2/24, 1:18 PM

No problem. It has only taken me two days

I think the issue here is that the queue is not a standard worker. It has extra limitations that nobody is mentioning anywhere. Something to add to the docs as it will save so much time

Poyo•12/2/24, 1:24 PM

By the way, when you run a browser in the queue directly, it keeps spinning until it exhausts all the CPU, and only after that does it time out. That is why we were seeing these random long waiting times. The browser blocks the consumer from execution until it times out, but it is my guess. I'm glad that I was able to help.

NNoGameNoFun hi, i trying to set the cpu ms limit for the worker that run as consumer of a qu...

Poyo•12/2/24, 1:40 PM

My assumption is that a queue consumer is not a standard worker. I would also use a Durable Object - basically a separate normal worker connected to your queue consumer

queues

PPoyo By the way, when you run a browser in the queue directly, it keeps spinning unti...

ajgeiss0702•12/2/24, 4:34 PM

I can basically confirm it is some issue with browser rendering + queues. The only thing I changed was moving the browser rendering logic to a DO and calling that instead of it just being a function in my consumer. The rest of the logic is the same, and I have not had the issue since making that change

Sami•12/3/24, 3:07 AM

Hi, is there a way to find how many outstanding items/messages are present in a queue? Especially, to use with pull-workers.

juanferreras•12/3/24, 1:51 PM

I thought queues was not working reliably because I was seeing some unprocessed rows in my database from time to time.
Actually, I had a

try/catch

try/catch

on my consumer worker so I was swallowing the errors myself haha. Turns out I was the unreliable one and queues works great

go•12/4/24, 10:19 AM

I am using a cron once a day to fetch items from my database, add them to a queue, and process the items. However, as the cron is adding tens of thousands of items to the queue, I am reaching the limit of worker invocations I can do (1000 per worker). What are recommended ways to handle this kind of large batch processing in Workers?

Ggnomesgames Hi! I’ve got a worker being called as a webhook from an external service (which ...

James•12/4/24, 3:00 PM

Nope, the consumer is run concurrently. You need a programmatic way to make these sequential as the more messages you queue, the single consumer scales up to consume on demand.

But if you chain messages via the queue, that would work.

Ggnomesgames Hi! I’ve got a worker being called as a webhook from an external service (which ...

Brett Willis•12/4/24, 6:18 PM

You can set max_concurrencymax_concurrency to

so that your consumer will be invoked only one at a time.
Each invocation can still of course contain a batch of multiple messages depending on your max_batch_sizemax_batch_size setting.
https://developers.cloudflare.com/queues/configuration/consumer-concurrency/

Cloudflare Docs

Consumer concurrency · Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.

Stefan Stührmann•12/4/24, 9:38 PM

Hey, is there any timeline when you plan to support running separate producer and consumer Workers bound to the same Queue locally? Usually you use queues to decouple your application therefore it is quite uncomfortable to not have the same level of local development on queues then on other Cloudflare components. And yes I'm already aware that I can trick this by "wiring" the consumer handler into the worker into the handler of the producer worker using the queue handler function out of the other worker and managing the consumer / producer mapping via wrangler.toml. It's just uncomfortable. (https://developers.cloudflare.com/queues/configuration/local-development/#known-issues)).

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as Cloudflare runs globally.

ajgeiss0702•12/4/24, 10:35 PM

im having a weird issue with one of my queues.
The queue itself is suddenly (as of this week) processing messages much slower than it used to. No changes have been pushed to the consumer at all in the past 7 months

Additionally, every time I try to look at the queue in the dashboard, the dashboard crashes

queue id is a11bfcc288144990acb3a9f0ba8c5adfa11bfcc288144990acb3a9f0ba8c5adf

ajgeiss0702•12/4/24, 10:36 PM

and this is what I mean by the dashboard crashing. This happens after clicking on the queue. I see the queue metrics page start to load, then it switches to this less than a second later

ajgeiss0702•12/4/24, 10:38 PM

and there is this error message in the browser console

Tilo•12/5/24, 12:16 PM

Hey, I have a app which uses browser rendering and queues in the same worker. So far I used wrangler dev --remote to develop the browser part, I wanted to add the queue but ran into the issue, that queues are not supported in remote dev. What is a possible solution here?

patotoma•12/6/24, 4:01 AM

Is there something wrong with Queues? Most of my messages are failing to deliver for what they normally works.

James•12/6/24, 4:46 AM

I cannot load any queues on my cloudflare dashboard, the dashboard site breaks when I click on any. Any help?

JJames I cannot load any queues on my cloudflare dashboard, the dashboard site breaks w...

Isaac McFadyen•12/6/24, 4:50 AM

Hey. Mind sending your account ID (it's fine to make it public, it's not secret info) and I can see if I can escalate this?

Isaac McFadyen•12/6/24, 4:51 AM

Given that a user was having the same issue the other day there's likely a bug in the Queues dashboard.

IIsaac McFadyen Hey. Mind sending your account ID (it's fine to make it public, it's not secret ...

James•12/6/24, 4:52 AM

sent thank you

JJames I cannot load any queues on my cloudflare dashboard, the dashboard site breaks w...

Isaac McFadyen•12/6/24, 3:42 PM

Hey - this is being investigated by the team, thanks for the report.

JJames I cannot load any queues on my cloudflare dashboard, the dashboard site breaks w...

Pranshu MaheshwariOP•12/6/24, 9:24 PM

We've found the issue here: the fix should roll out soon!

Walshy•12/6/24, 9:52 PM

Should be out and fixed

kimjongillest•12/12/24, 3:01 AM

is the best way to segregate HTTP access for queues to manage secrets within a worker?

eg: client calls a normal worker rest endpoint and passes the secret, then the worker does a switch statement to correctly forward the message to a queue while checking the secret against that queue name using a map?

Aries•12/17/24, 6:44 PM

Is there a way to de-dupe queue items, ie, have a unique-by key, which will ensure it is in the queue only once?

Aries•12/17/24, 6:45 PM

optimally, you'd be able to specific a key when writing a message.

sup filistine•1/5/25, 1:55 AM

maybe a short question on queues: x-posting from workers-help: workers-and-pages-helpQueues are not scaling up, consumer wise when using a regular (non pull consumer)

There's a few reasons why consumers might not autoscale (documented here: https://developers.cloudfl

Similar Threads

Similar Threads

Similar Threads