I was wondering. if I need to send a daily statement to a lot of users, can queues help me with this

I was wondering. if I need to send a daily statement to a lot of users, can queues help me with this?

PPato I was wondering. if I need to send a daily statement to a lot of users, can queu...

John Spurlock•12/20/22, 5:51 PM

absolutely, insert the work with a daily

cron

cron

- either as separate messages or one large message, then make incremental progress inside the queuequeue handler

PatoOP•12/20/22, 5:53 PM

And by progress inside the queue handler you mean using a ConsumerConsumer?

PPato And by progress inside the queue handler you mean using a `Consumer`?

John Spurlock•12/20/22, 5:54 PM

yes, once you designate a worker as the queue consumer, it will call your queuequeue handler for any "work items" (messages of reports to send) https://developers.cloudflare.com/queues/javascript-apis/#consumer

PatoOP•12/20/22, 5:54 PM

So you would recommend to have 2 separate worker right?

John Spurlock•12/20/22, 5:54 PM

keep in mind the cpu limits and other limits still apply, so you'll need to break it up into smaller items

PatoOP•12/20/22, 5:55 PM

one that adds to the queue and one that only reads from it?

John Spurlock•12/20/22, 5:55 PM

personally I just have one, a worker can produce items and consume them

PatoOP•12/20/22, 5:55 PM

I see

John Spurlock•12/20/22, 5:55 PM

no need to overcomplicate unless the codebases are really different (and require different scripts)

PatoOP•12/20/22, 5:56 PM

What would be the best practice for this use case:
- My app produces daily statements (for some clients, not necessarily all of them)
- I need to fetch those statements from the accounts that have them and send them by email

John Spurlock•12/20/22, 6:01 PM

without knowing much about your setup, at a high level: have

cron

cron

calculate and materialize the work for the day, publish at least one queue msg when done, then handle/send those reports in the queuequeue handler. You'll need to decide whether to keep all the state in the messages themselves, or use storage (like in a DO) - so you don't send the same email twice (queues can sometimes send you the same message twice)

John Spurlock•12/20/22, 6:03 PM

if it were me, i'd go with the latter: keep the work-items in a DO, state for clientid+reportday+emailstatus, then just have messages that keep making progress on that list till it's complete

John Spurlock•12/20/22, 6:07 PM

could use R2 for the state too, or if you have an existing system db external to CF that would work too

kian•12/20/22, 6:18 PM

changes and lastRowId are unimplemented - at least last I saw on Nov 9th

kian•12/20/22, 6:18 PM

Was you seeing anything in that property recently?

JJohn Spurlock without knowing much about your setup, at a high level: have `cron` calculate an...

PatoOP•12/20/22, 6:23 PM

I see, thanks! This was very helpful

kian•12/20/22, 7:24 PM

You'd need to be a consumer for the dlqdlq queue for those messages to go to your async queueasync queue as far as I can tell

JJohand Yeah, I tried something like this too. Can a worker only subscribe to one queue?

Matt Silverlock•12/20/22, 11:44 PM

What is the name of your DLQ? You have to subscribe to it as well — from the same Worker or another. See https://developers.cloudflare.com/queues/learning/how-queues-works/#consumers

How Queues Works · Cloudflare Queues

Reliably send and receive messages without the egress fees.

Matt Silverlock•12/20/22, 11:44 PM

And https://developers.cloudflare.com/queues/configuration/#consumer

Cloudflare Queues - Configuration · Cloudflare Queues

Reliably send and receive messages without the egress fees.

JJohand But something weird is happening. Even when I set `max_retries = 10` it only ret...

Emilio•12/21/22, 6:13 PM

When running locally, you're not hitting the Cloudflare edge network, but rather miniflare (https://miniflare.dev/). Miniflare is a simulated Worker environment. It's possible that Miniflare is not picking up the change in your consumer configuration. Let me see if I can reproduce.

UPDATE: Was able to reproduce, looks like miniflare doesn't honor the retry count configured (always uses 3) -- and that's something we can fix

SSibi Hey everyone! We just went live with Queues in production yesterday for batching...

Matt Silverlock•12/22/22, 3:57 PM

Looking into it. Can you ping me your account ID?

SSibi Thanks for the quick response! Sent it via DM

Matt Silverlock•12/22/22, 5:41 PM

Related to this for others: we had a regression with how timestamps were handled internally, that was leading to messages being marked "ready to send" sooner than they actually were. All new messages should be batched correctly now.

If you have/had a backlog, those messages will need to be processed first. Existing messages may be sent too eagerly until your consumer has caught up.

Unsmart•12/23/22, 3:07 AM

You could upload a file to R2 and send a key reference in the message

Unsmart•12/23/22, 3:30 AM

It depends on your upload speed but within the CF network probably not too much maybe like 1-2s depending if the bucket is in the same region

Unsmart•12/23/22, 4:24 AM

There is no colo nearest the dashboard it's edge. They would both do exactly the same thing.

Unsmart•12/23/22, 4:42 AM

That's about what I get

Unsmart•12/23/22, 4:43 AM

And don't think it's possible to know the colo your queue is in

GGary Somerhalder Interesting idea

aarhus•12/23/22, 11:56 AM

Or KV - but you would need to handle the eventual consistency (shouldn't be an issue if you create a unique ID for each object, but your consumer needs to handle it not being present)

UUnsmart There is no colo nearest the dashboard it's edge. They would both do exactly the...

Hello, I’m Allie!•12/23/22, 1:26 PM

Though sometimes, like for D1, if you do it from the dash, it is routed through the API, which I believe runs in IAD.

RRob > is that normal for now, yes. look for significant improvements here as the bet...

John Spurlock•12/23/22, 4:00 PM

Rob, it's an interesting question about the colo - is it safe to say the queue "resides" in the colo that the queuequeue handler runs in? Ok only mildly interesting for minimizing latency from publishers I guess. Ok only interesting to plot out a Dr Strangelove style map of the architecture...

JJohn Spurlock Rob, it's an interesting question about the colo - is it safe to say the queue "...

Emilio•12/23/22, 5:47 PM

A queue resides in a particular colo, yes. This is also something you can expect us to refine as we get closer to GA. Not saying it's going to change, but it's definitely something we are thinking about.

EEmilio A queue resides in a particular colo, yes. This is also something you can expect...

John Spurlock•12/23/22, 5:58 PM

is it the same colo as where the queuequeue handler runs?

I was wondering. if I need to send a daily statement to a lot of users, can queues help me with this

Similar Threads