im also running into a weird issue where batches are always taking 15 minutes, hitting the wall time

im also running into a weird issue where batches are always taking 15 minutes, hitting the wall time limit (which in the dashboard shows up as a cpu limit error for some reason, even though cpu times stay around 300ms per batch), even though everything finishes way before then, which causes the backlog to take even longer to get through

queue id is

d1a94a7c4e4a4c23a6dbc74a8abbad11

d1a94a7c4e4a4c23a6dbc74a8abbad11

d1a94a7c4e4a4c23a6dbc74a8abbad11

d1a94a7c4e4a4c23a6dbc74a8abbad11 if it helps

ajgeiss0702OP•11/25/24, 6:53 PM

please ping me if you reply

Aajgeiss0702 im also running into a weird issue where batches are always taking 15 minutes, h...

ajgeiss0702OP•11/25/24, 10:03 PM

i suspect this has something to do with some request not being completed for some reason, which I was having issues with in the past. Is there a way to tell what it is waiting for?

oof2win2•11/26/24, 12:25 AM

is there a way to limit queue throughput? i have an upstream rate limit that i don't want to hit. i might have event spikes during the day and i don't want to drop requests so i want to limit myself to the set throughput (relatively speaking it's low, otherwise i wouldn't be asking for this)

oof2win2•11/26/24, 12:27 AM

like i want at most 10 messages per second. i have max_concurrencymax_concurrency on 1 but if the 10 messages in the batch complete faster than in 1 second then it doesn't work

Ooof2win2 like i want at most 10 messages per second. i have `max_concurrency` on 1 but if...

ajgeiss0702OP•11/26/24, 5:55 AM

you could simply wait a second after each message in the batch. Then if you have all 10 messages in a batch, then it shouldn't complete more than 1 per second

ajgeiss0702OP•11/26/24, 5:56 AM

for example with this function, just call await wait(1000)await wait(1000) to wait a second

ajgeiss0702OP•11/26/24, 5:56 AM

this does not count as cpu time

ajgeiss0702OP•11/26/24, 5:57 AM

you could of course decrease that wait period if you want to process message faster than one per second (you could use 100ms to process at most 10 per second)

Aajgeiss0702 im also running into a weird issue where batches are always taking 15 minutes, h...

ac•11/26/24, 3:21 PM

I am seeing the same thing both with low concurrency and timeouts

ac•11/26/24, 3:22 PM

So the queue backlog is huge

Pranshu Maheshwari•11/27/24, 3:37 PM

There's a few reasons why consumers might not autoscale (documented here: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/#why-are-my-consumers-not-autoscaling) (cc @ajgeiss0702)

If you're seeing timeouts @ac, it most likely means your Queue consumer is taking too long to process a message. You'll need to refactor your consumer to process the messages in under 30s of CPU time

Cloudflare Docs

Consumer concurrency | Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.

PPranshu Maheshwari There's a few reasons why consumers might not autoscale (documented here: https:...

ac•11/27/24, 3:39 PM

Thanks, I will take a look. Ultimately I believe the issue is browser rendering limits/the CF Browser Renderer holding onto processes for way longer than it's supposed to, which causes the whole queue to be backed up

PPranshu Maheshwari There's a few reasons why consumers might not autoscale (documented here: https:...

ajgeiss0702OP•11/27/24, 6:25 PM

I saw this, and I don't think it applies:
1. Max concurrency is not set
2. No errors are reported in the queues dashboard
3. batches are processed every 15 minutes (because of the other issue I mentioned) and it hasn't really scaled up

It has scaled up to 2 concurrency a few times, but hasn't gone past that, and it didnt stay on 2 for very long

xiechao•11/30/24, 6:55 AM

I would like to ask about billing related issues. I am currently facing a problem

xiechao•11/30/24, 6:56 AM

Remaining time on 11 × Images Stream Bundle Basic Stream storage per thousand minutes

xiechao•11/30/24, 6:56 AM

What is the fee?

1984 Ford Laser•11/30/24, 8:14 AM

?crossposting

SuperHelpflare•11/30/24, 8:14 AM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Poyo•12/1/24, 1:10 PM

I'm trying to implement a simple queue. My worker sends a queue message to the consumer. I can see the message in the queue via the dashboard, but for some reason, the messages are just sitting in the queue, and processing doesn't start for 20 minutes. Do you have any ideas about what can cause this? I have logging and I can see that it is just sitting in the queue - no errors or anything.

Poyo•12/1/24, 1:12 PM

The timeouts and settings are default. I've spent a couple of hours but can't find a solution. I hope it is not an unannounced service disruption.

Poyo•12/1/24, 1:23 PM

Now it took a message to start processing - 4m 38s

Poyo•12/1/24, 6:18 PM

Okay, I found a problem - the queue exceeded the CPU limit when the puppeteer was imported.

Poyo•12/1/24, 10:37 PM

Found a way to fix it - don't use the browser inside queue but move it to a separate worker. Check the docs for Rendering Worker with Durable Object

ajgeiss0702OP•12/2/24, 12:45 AM

i wonder if thats the fix to my problem

ajgeiss0702OP•12/2/24, 3:57 AM

wow i think that might have solved my problem too

ajgeiss0702OP•12/2/24, 3:57 AM

so it seems like browser rendering has some issue with queues for some reason

ajgeiss0702OP•12/2/24, 3:58 AM

would have been nice to know this before black friday week, but better late than never

ajgeiss0702OP•12/2/24, 4:04 AM

its the first time ive seen this number in weeks! so beautiful!

PPoyo Found a way to fix it - don't use the browser inside queue but move it to a sepa...

ajgeiss0702OP•12/2/24, 4:11 AM

thank you very much. You just solved an issue i've been having for weeks and could not find a solution for!

NoGameNoFun•12/2/24, 6:54 AM

hi, i trying to set the cpu ms limit for the worker that run as consumer of a queueu, based on the documentation, it suppose can set a max limit of 15 minutes of cpu run limit, but when i try to set via wrangler or even using the dashboard, both i got is max 30,000 ms only

Aajgeiss0702 thank you very much. You just solved an issue i've been having for weeks and cou...

Poyo•12/2/24, 1:18 PM

No problem. It has only taken me two days

I think the issue here is that the queue is not a standard worker. It has extra limitations that nobody is mentioning anywhere. Something to add to the docs as it will save so much time

Poyo•12/2/24, 1:24 PM

By the way, when you run a browser in the queue directly, it keeps spinning until it exhausts all the CPU, and only after that does it time out. That is why we were seeing these random long waiting times. The browser blocks the consumer from execution until it times out, but it is my guess. I'm glad that I was able to help.

NNoGameNoFun hi, i trying to set the cpu ms limit for the worker that run as consumer of a qu...

Poyo•12/2/24, 1:40 PM

My assumption is that a queue consumer is not a standard worker. I would also use a Durable Object - basically a separate normal worker connected to your queue consumer

queues

PPoyo By the way, when you run a browser in the queue directly, it keeps spinning unti...

ajgeiss0702OP•12/2/24, 4:34 PM

I can basically confirm it is some issue with browser rendering + queues. The only thing I changed was moving the browser rendering logic to a DO and calling that instead of it just being a function in my consumer. The rest of the logic is the same, and I have not had the issue since making that change

Sami•12/3/24, 3:07 AM

Hi, is there a way to find how many outstanding items/messages are present in a queue? Especially, to use with pull-workers.

juanferreras•12/3/24, 1:51 PM

I thought queues was not working reliably because I was seeing some unprocessed rows in my database from time to time.
Actually, I had a

try/catch

try/catch

on my consumer worker so I was swallowing the errors myself haha. Turns out I was the unreliable one and queues works great

go•12/4/24, 10:19 AM

I am using a cron once a day to fetch items from my database, add them to a queue, and process the items. However, as the cron is adding tens of thousands of items to the queue, I am reaching the limit of worker invocations I can do (1000 per worker). What are recommended ways to handle this kind of large batch processing in Workers?

Ggnomesgames Hi! I’ve got a worker being called as a webhook from an external service (which ...

James•12/4/24, 3:00 PM

Nope, the consumer is run concurrently. You need a programmatic way to make these sequential as the more messages you queue, the single consumer scales up to consume on demand.

But if you chain messages via the queue, that would work.

Ggnomesgames Hi! I’ve got a worker being called as a webhook from an external service (which ...

Brett Willis•12/4/24, 6:18 PM

You can set max_concurrencymax_concurrency to

so that your consumer will be invoked only one at a time.
Each invocation can still of course contain a batch of multiple messages depending on your max_batch_sizemax_batch_size setting.
https://developers.cloudflare.com/queues/configuration/consumer-concurrency/

Cloudflare Docs

Consumer concurrency · Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.

Stefan Stührmann•12/4/24, 9:38 PM

Hey, is there any timeline when you plan to support running separate producer and consumer Workers bound to the same Queue locally? Usually you use queues to decouple your application therefore it is quite uncomfortable to not have the same level of local development on queues then on other Cloudflare components. And yes I'm already aware that I can trick this by "wiring" the consumer handler into the worker into the handler of the producer worker using the queue handler function out of the other worker and managing the consumer / producer mapping via wrangler.toml. It's just uncomfortable. (https://developers.cloudflare.com/queues/configuration/local-development/#known-issues)).

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as Cloudflare runs globally.

ajgeiss0702OP•12/4/24, 10:35 PM

im having a weird issue with one of my queues.
The queue itself is suddenly (as of this week) processing messages much slower than it used to. No changes have been pushed to the consumer at all in the past 7 months

Additionally, every time I try to look at the queue in the dashboard, the dashboard crashes

queue id is a11bfcc288144990acb3a9f0ba8c5adfa11bfcc288144990acb3a9f0ba8c5adf

ajgeiss0702OP•12/4/24, 10:36 PM

and this is what I mean by the dashboard crashing. This happens after clicking on the queue. I see the queue metrics page start to load, then it switches to this less than a second later

ajgeiss0702OP•12/4/24, 10:38 PM

and there is this error message in the browser console

Tilo•12/5/24, 12:16 PM

Hey, I have a app which uses browser rendering and queues in the same worker. So far I used wrangler dev --remote to develop the browser part, I wanted to add the queue but ran into the issue, that queues are not supported in remote dev. What is a possible solution here?

patotoma•12/6/24, 4:01 AM

Is there something wrong with Queues? Most of my messages are failing to deliver for what they normally works.

James•12/6/24, 4:46 AM

I cannot load any queues on my cloudflare dashboard, the dashboard site breaks when I click on any. Any help?

JJames I cannot load any queues on my cloudflare dashboard, the dashboard site breaks w...

Isaac McFadyen•12/6/24, 4:50 AM

Hey. Mind sending your account ID (it's fine to make it public, it's not secret info) and I can see if I can escalate this?

Isaac McFadyen•12/6/24, 4:51 AM

Given that a user was having the same issue the other day there's likely a bug in the Queues dashboard.

im also running into a weird issue where batches are always taking 15 minutes, hitting the wall time

Similar Threads

Similar Threads

Similar Threads