And by progress inside the queue handler you mean using a `Consumer`? - Cloudflare Developers

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

Pato

PPato And by progress inside the queue handler you mean using a `Consumer`?

John Spurlock

J

John Spurlock•12/20/22, 5:54 PM

yes, once you designate a worker as the queue consumer, it will call your queuequeue handler for any "work items" (messages of reports to send) https://developers.cloudflare.com/queues/javascript-apis/#consumer

Pato

P

PatoOP•12/20/22, 5:54 PM

So you would recommend to have 2 separate worker right?

John Spurlock

J

John Spurlock•12/20/22, 5:54 PM

keep in mind the cpu limits and other limits still apply, so you'll need to break it up into smaller items

Pato

P

PatoOP•12/20/22, 5:55 PM

one that adds to the queue and one that only reads from it?

John Spurlock

J

John Spurlock•12/20/22, 5:55 PM

personally I just have one, a worker can produce items and consume them

Pato

P

PatoOP•12/20/22, 5:55 PM

I see

John Spurlock

J

John Spurlock•12/20/22, 5:55 PM

no need to overcomplicate unless the codebases are really different (and require different scripts)

Pato

P

PatoOP•12/20/22, 5:56 PM

What would be the best practice for this use case:
- My app produces daily statements (for some clients, not necessarily all of them)
- I need to fetch those statements from the accounts that have them and send them by email

John Spurlock

J

John Spurlock•12/20/22, 6:01 PM

without knowing much about your setup, at a high level: have

cron

cron

calculate and materialize the work for the day, publish at least one queue msg when done, then handle/send those reports in the queuequeue handler. You'll need to decide whether to keep all the state in the messages themselves, or use storage (like in a DO) - so you don't send the same email twice (queues can sometimes send you the same message twice)

John Spurlock

J

John Spurlock•12/20/22, 6:03 PM

if it were me, i'd go with the latter: keep the work-items in a DO, state for clientid+reportday+emailstatus, then just have messages that keep making progress on that list till it's complete

John Spurlock

J

John Spurlock•12/20/22, 6:07 PM

could use R2 for the state too, or if you have an existing system db external to CF that would work too

kian

K

kian•12/20/22, 6:18 PM

changes and lastRowId are unimplemented - at least last I saw on Nov 9th

kian

K

kian•12/20/22, 6:18 PM

Was you seeing anything in that property recently?

John Spurlock

JJohn Spurlock without knowing much about your setup, at a high level: have `cron` calculate an...

Pato

P

PatoOP•12/20/22, 6:23 PM

I see, thanks! This was very helpful

kian

K

kian•12/20/22, 7:24 PM

You'd need to be a consumer for the dlqdlq queue for those messages to go to your async queueasync queue as far as I can tell

Johand

JJohand Yeah, I tried something like this too. Can a worker only subscribe to one queue?

Matt Silverlock

M

Matt Silverlock•12/20/22, 11:44 PM

What is the name of your DLQ? You have to subscribe to it as well — from the same Worker or another. See https://developers.cloudflare.com/queues/learning/how-queues-works/#consumers

How Queues Works · Cloudflare Queues

Reliably send and receive messages without the egress fees.

How Queues Works · Cloudflare Queues

Matt Silverlock

M

Matt Silverlock•12/20/22, 11:44 PM

And https://developers.cloudflare.com/queues/configuration/#consumer

Cloudflare Queues - Configuration · Cloudflare Queues

Reliably send and receive messages without the egress fees.

Cloudflare Queues - Configuration · Cloudflare Queues

Johand

JJohand But something weird is happening. Even when I set `max_retries = 10` it only ret...

Emilio

E

Emilio•12/21/22, 6:13 PM

When running locally, you're not hitting the Cloudflare edge network, but rather miniflare (https://miniflare.dev/). Miniflare is a simulated Worker environment. It's possible that Miniflare is not picking up the change in your consumer configuration. Let me see if I can reproduce.

UPDATE: Was able to reproduce, looks like miniflare doesn't honor the retry count configured (always uses 3) -- and that's something we can fix

Sibi

SSibi Hey everyone! We just went live with Queues in production yesterday for batching...

Matt Silverlock

M

Matt Silverlock•12/22/22, 3:57 PM

Looking into it. Can you ping me your account ID?

Sibi

SSibi Thanks for the quick response! Sent it via DM

Matt Silverlock

M

Matt Silverlock•12/22/22, 5:41 PM

Related to this for others: we had a regression with how timestamps were handled internally, that was leading to messages being marked "ready to send" sooner than they actually were. All new messages should be batched correctly now.

If you have/had a backlog, those messages will need to be processed first. Existing messages may be sent too eagerly until your consumer has caught up.

Unsmart

U

Unsmart•12/23/22, 3:07 AM

You could upload a file to R2 and send a key reference in the message

Unsmart

U

Unsmart•12/23/22, 3:30 AM

It depends on your upload speed but within the CF network probably not too much maybe like 1-2s depending if the bucket is in the same region

Unsmart

U

Unsmart•12/23/22, 4:24 AM

There is no colo nearest the dashboard it's edge. They would both do exactly the same thing.

Unsmart

U

Unsmart•12/23/22, 4:42 AM

That's about what I get

Unsmart

U

Unsmart•12/23/22, 4:43 AM

And don't think it's possible to know the colo your queue is in

Gary Somerhalder

GGary Somerhalder Interesting idea

aarhus

A

aarhus•12/23/22, 11:56 AM

Or KV - but you would need to handle the eventual consistency (shouldn't be an issue if you create a unique ID for each object, but your consumer needs to handle it not being present)

Unsmart

UUnsmart There is no colo nearest the dashboard it's edge. They would both do exactly the...

Hello, I’m Allie!

H

Hello, I’m Allie!•12/23/22, 1:26 PM

Though sometimes, like for D1, if you do it from the dash, it is routed through the API, which I believe runs in IAD.

Rob

RRob > is that normal for now, yes. look for significant improvements here as the bet...

John Spurlock

J

John Spurlock•12/23/22, 4:00 PM

Rob, it's an interesting question about the colo - is it safe to say the queue "resides" in the colo that the queuequeue handler runs in? Ok only mildly interesting for minimizing latency from publishers I guess. Ok only interesting to plot out a Dr Strangelove style map of the architecture...

John Spurlock

JJohn Spurlock Rob, it's an interesting question about the colo - is it safe to say the queue "...

Emilio

E

Emilio•12/23/22, 5:47 PM

A queue resides in a particular colo, yes. This is also something you can expect us to refine as we get closer to GA. Not saying it's going to change, but it's definitely something we are thinking about.

Emilio

EEmilio A queue resides in a particular colo, yes. This is also something you can expect...

John Spurlock

J

John Spurlock•12/23/22, 5:58 PM

is it the same colo as where the queuequeue handler runs?