I'm halfway finished with the implementation of local queues using durable objects in miniflare. Thi

I'm halfway finished with the implementation of local queues using durable objects in miniflare. This will allow you to have local persistent queues that can be shared between multiple workers instead of the in memory array based queues and therefore you can test distributed / microservice architectures using queues fully locally. May I ask why you need --remote? For me the biggest issue is the lack of local queues between multiple workers (the main purpose of queues).

SStefan Stührmann I'm halfway finished with the implementation of local queues using durable objec...

Mitya•1/30/25, 5:32 PM

Thanks. I need remote for two reasons, mainly. Firstly, and for some unknown reason, my Worker refuses to connect to my remote PG DB (even though I can connect to it on my machine via terminal). The only workaround was to use the remote flag and access via a hyperdrive. This seems to be a problem unique to me and my DNS or network or whatever. Secondly, I use CF AI.

Mitya•1/30/25, 5:33 PM

Is there a preferred way to simulate queues locally with the remote flag? Sort of like the __schedule__schedule query string arg for testing CRON schedules locally.

Mitya•1/31/25, 10:49 PM

Question. The docs say consumers get 30 seconds of CPU time. I'm trying to understand what this means practically. If my consumer is sent a batch of 10 messages, and each of those messages needs sending to e.g. an AI analysis API, is that likely to max out the 30 seconds of CPU time? How can one reliably predict CPU time for a given task?

MMitya Question. The docs [say](https://developers.cloudflare.com/queues/platform/limit...

Isaac McFadyen•1/31/25, 10:52 PM

?cpu-time

SuperHelpflare•1/31/25, 10:52 PM

CPU Time: the amount of work the CPU needs to do, measured in msms. More CPU-heavy things will increase it (for example, hashing passwords). This is not the same as the time a

fetch

fetch

request takes.
Wall Time: the actual amount of time a Worker runs for, as though you measured it with a clock or your browser DevTools

Isaac McFadyen•1/31/25, 10:52 PM

CPU time is not wall time.

Isaac McFadyen•1/31/25, 10:52 PM

If you're making external requests then you're unlikely to get even near 30s.

Isaac McFadyen•1/31/25, 10:52 PM

More likely you'd be in the tens of ms total.

Rahul gangotri•2/1/25, 6:29 AM

Hi
I have deployed a queue that does the website crawling and have added many console logs

problem is that browser rendering only works with

 wrangler dev --remote

 wrangler dev --remote

but queue doesn't support it

if i am deploying its not showing my console, how to test it?

Rahul gangotri•2/1/25, 6:30 AM

i want to reuse browser session so i think queue is the best way, or can i use workflow also?

Rahul gangotri•2/1/25, 6:30 AM

i am crawling in a nested way to fetch the content of all provided links

RRahul gangotri Hi I have deployed a queue that does the website crawling and have added many co...

Hello, I’m Allie!•2/1/25, 6:33 AM

You can deploy it

Hello, I’m Allie!•2/1/25, 6:33 AM

That’s about it when you have a conflict like that

Rahul gangotri•2/1/25, 6:39 AM

If i deploy, how do i check my console logs for debugging?

Rahul gangotri•2/1/25, 6:40 AM

Logs page of worker does not shows console and breaks all the time

Rahul gangotri•2/1/25, 6:40 AM

How can i see logs to debug what’s wrong?

HHello, I’m Allie!You can deploy it

Rahul gangotri•2/1/25, 6:42 AM

Tell mw one thing

To reuse sessions and to overcome the 10 session limit

Can i use durable object? Or workflow? Or queue is the best way?

Rahul gangotri•2/1/25, 6:43 AM

Because if all sessions are being used i need to wait and retry

RRahul gangotri How can i see logs to debug what’s wrong?

Hello, I’m Allie!•2/1/25, 8:03 AM

wrangler tailwrangler tail?

Rahul gangotri•2/1/25, 8:05 AM

it does not provide console.log 's

Rahul gangotri•2/1/25, 8:07 AM

even i tried connecting to baselime, it only show basic things like incoming request to queue endpoint and when queue is picked by consumer

that's all

no console logs nothing

o7•2/1/25, 11:03 AM

can Queues be used to batch cache purge a bunch of urls

Oo7 can Queues be used to batch cache purge a bunch of urls

Mitya•2/1/25, 11:51 AM

Sure. The producer worker pushes the URLs into the queue, the consumer worker receives a batch of them, and then calls one of the cache purge API endpoints.

thehungrybird_•2/1/25, 3:16 PM

guys, I'm building a small application with hono.js, but fetch with post method is not working in queues, can anyone help?

RRahul gangotri even i tried connecting to baselime, it only show basic things like incoming req...

aarhus•2/1/25, 9:35 PM

@Rahul gangotri use a service binding to another worker? The other worker has the logging in. I think this works

Aaarhus @Rahul gangotri use a service binding to another worker? The other worker has th...

Rahul gangotri•2/2/25, 5:02 AM

I tried tail worker setup, not printing console logs

Rahul gangotri•2/2/25, 5:03 AM

I think only queue consumer is not sending logs

pat•2/2/25, 8:17 AM

I can also confirm that getting logs out of a queue consumer can be a bit moody. It works sometimes, not other times. Seems to come through immediately for requests to

fetch

fetch

though.

pat•2/2/25, 10:22 PM

Hey guys, is there any problem with recursive queue consumers/producers? I have a consumer that produces to the same queue it consumes from and it seems like metrics are significantly delayed. That said though, my queue jobs do take ~30s to run each and can only run one at a time by design. Perhaps it takes some time for a new deployment to be hooked up as I'm testing some stuff and deploying pretty often. I can see that the queue I am using is increasing in size and my db is getting new data so it is consuming and producing. It seems that as soon as I stop producing to the same queue I am consuming from, the metrics return

Another question that I am struggling to find any information on is how to interpret the execution duration GB-sec value on workers. What does this mean?

pat•2/2/25, 10:45 PM

Also is there a way to flush a queue? My queue grows much quicker than I consume it so once I have enough data I'd like to just flush all the messages to save the cpu time of consuming to ignore them

EndiM•2/6/25, 7:26 PM

Hello. Is there an issue on the queues right now? Our queue has around 500 messages in the backlog. Started 15 minutes ago.

EndiM•2/6/25, 7:30 PM

Just started processing again.

Sam•2/10/25, 11:12 PM

If I want to be able to queue a bunch of items (say 10,000) but have a certain amount go out per day (say 1,000) how could I achieve this?

SSam If I want to be able to queue a bunch of items (say 10,000) but have a certain a...

James•2/11/25, 4:57 PM

cron worker feeds queue in batches

JJames cron worker feeds queue in batches

Sam•2/11/25, 4:59 PM

The master list of jobs to feed into the cron would you suggest using like an r2 or d1 or something? Then mark each as processed or something once each is done.

Brett Willis•2/12/25, 2:05 AM

Feature request: would be nice to be able to message set expiry (max age and/or max retries) at the message level.
We have a scenario where some messages are important to be processed eventually, and some messages should be discarded if not processed within a certain timeframe.

EndiM•2/13/25, 9:06 PM

Hello. What would the average queue backlog be, on a healthy queue? Our right now keeps on being 760, with a delayed backlog of around 530..

sathoro•2/14/25, 7:09 AM

if you need more concurrency, create your own queue system with Durable Objects so it can be scaled infinitely

Pranshu Maheshwari•2/14/25, 8:36 PM

A quick feature update to share: you can now customize the retention period for a queue https://developers.cloudflare.com/changelog/2025-02-14-customize-queue-retention-period/

Cloudflare Docs

Customize queue message retention periods

achesui•2/15/25, 1:56 PM

Hello, is there a way to bind a queue Event Notification to a local R2 bucket?. For development

scotto•2/17/25, 11:56 AM

setting this in queues consumer

[[queues.consumers]]
queue = "gopersonal-llm-ranking-queue"
max_batch_size = 100
max_batch_timeout = 180
max_concurrency = 1

[[queues.consumers]]
queue = "gopersonal-llm-ranking-queue"
max_batch_size = 100
max_batch_timeout = 180
max_concurrency = 1

and getting the error

✘ [ERROR] A request to the Cloudflare API (/accounts/920b1a6e159cf77dab28969103a4765b/queues/c30e85e6726c455b880498715d8a0b4c/consumers/5bad5c90b2644f498177c45a164e96ac) failed.

Queue consumer (type worker) has invalid settings: maximum wait time must be between 0 and 60000
ms. [code: 100127]

If you think this is a bug, please open an issue at:
https://github.com/cloudflare/workers-sdk/issues/new/choose

if we put the value of max_batch_timeoutmax_batch_timeout to 60 it works.., that means the error message is in milliseconds and the actual limit is 60 seconds?

GitHub

Build software better, together

GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.

Sscotto setting this in queues consumer ```[[queues.consumers]] queue = "gopersonal-llm-...

John Spurlock•2/17/25, 3:56 PM

yep you got it, the max batch wait time is 60 seconds [1], the backend rest api takes milliseconds [2], so wrangler multiplies the config value by 1000 [3], making the error message confusing : )

[1] https://developers.cloudflare.com/queues/platform/limits/
[2] https://developers.cloudflare.com/api/resources/queues/subresources/consumers/methods/create/
[3] https://github.com/cloudflare/workers-sdk/blob/0322d085f634c1a0a12a59b4db293088d0cadb62/packages/wrangler/src/deploy/deploy.ts#L1236

JJohn Spurlock yep you got it, the max batch wait time is 60 seconds [1], the backend rest api ...

scotto•2/17/25, 3:56 PM

thanks, any plans to increase those limits?

Sscotto thanks, any plans to increase those limits?

John Spurlock•2/17/25, 4:05 PM

someone orange would have to speak to that - personally I would love to see larger values allowed there and also the batch size 100msg/256kb lifted a bit as well

HowlOftheSun•2/17/25, 11:28 PM

~~What would cause a queue to send the same msg multiple times? Is there a timeout i might have set wrong?~~ Errors cause retries....

which cause more errors

ajgeiss0702•2/23/25, 9:09 PM

Did the queues sendBatchsendBatch api change?
I've been running a worker in production for around 7 months now, and it has been working fine.
The past few days (first occurance was Febuary 21st, 23:40 UTC, and the previous successful evocation was 23:10 UTC), i've suddenly been getting an error: sendBatch() requires at least one messagesendBatch() requires at least one message

I've added an additional check in my code to make sure the batch isnt empty to fix the issue, but I missed 2 days of data (that I cannot recover) before I noticed the issue.

Is this a recent change in the queues api? I would appreciate it if you could ping me when you reply

adeola13.•2/24/25, 2:35 AM

Hello! How does one deploy multiple queue handlers (for different queues) in one single worker deployment project in wrangler?

So in my main index.ts I would like to have multiple queue consumer handlers for different queues.

import producerHandler from './producer';
import consumerHandler from './consumer';
import { Env, TrackingEvent } from './types';

export default {
    fetch: producerHandler.fetch,
    queue: consumerHandler.queue,
} satisfies ExportedHandler<Env, TrackingEvent>;

import producerHandler from './producer';
import consumerHandler from './consumer';
import { Env, TrackingEvent } from './types';

export default {
    fetch: producerHandler.fetch,
    queue: consumerHandler.queue,
} satisfies ExportedHandler<Env, TrackingEvent>;

Aadeola13.Hello! How does one deploy multiple queue handlers (for different queues) in one...

James•2/24/25, 3:52 AM

You can only have one consumer function in your worker code, but you can get the name of the queue on MessageBatchMessageBatch: https://developers.cloudflare.com/queues/configuration/javascript-apis/#messagebatch

This way you can effectively handle multiple different queues and just change the code you execute depending on the queue name

JJames You can only have one consumer function in your worker code, but you can get the...

adeola13.•2/24/25, 3:56 AM

ah I see! Very helpful! And so then I can just bind multiple queue consumers to this worker and then my handler would only be able to read from messages send to the consumers that were bound from it is that right?

I'm halfway finished with the implementation of local queues using durable objects in miniflare. Thi

Similar Threads

Similar Threads

Similar Threads