Cloudflare Developers•4y ago

Is there a way to see the queue length?

KKai Is there a way to see the queue length?

John Spurlock•12/15/22, 4:23 PM

Not yet: queues

KaiOP•12/15/22, 4:36 PM

unfortunate, thanks

KaiOP•12/15/22, 5:35 PM

Awesome to see this happening

It's not really important to me, just interesting, so happy to wait until it's implemented normally

James•12/15/22, 9:16 PM

yeah with wrangler dev --localwrangler dev --local

KaiOP•12/16/22, 1:29 AM

Note that local queues are all in memory though, no persistence support from what I can tell and it can pretty easily run OOM.

Jjohnyukon21 Is it only possible to write to and read from queues from the context of workers...

John Spurlock•12/17/22, 7:00 PM

workers only at the moment, although http endpoints are planned: https://developers.cloudflare.com/queues/javascript-apis/

In the future, we expect to support other APIs, such as HTTP endpoints to send or receive messages.

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Reliably send and receive messages without the egress fees.

Method•12/18/22, 2:41 AM

not clear to me, how is pubsub any different than queues? you send and receive messages to/from a subscriber/publisher...

Unsmart•12/18/22, 2:46 AM

Queues are 1 to 1 and pubsub is 1 to many

Unsmart•12/19/22, 9:37 PM

batching data, reducing load on databases or other apis that cant handle the raw load

zegevlier•12/20/22, 6:40 AM

And automatic retry on fail ^

Aaltryne I have a GPU process that can handle 1 task at a time. The tasks are 15s - 3 min...

Matt Silverlock•12/20/22, 1:23 PM

There is no consumer concurrency right now. You’ll get a batch up to that maximum size, and you won’t get another batch until your consumer processes (success or failure) the batch.

Matt Silverlock•12/20/22, 1:24 PM

Consumer concurrency is coming, but just “turning it on by default” would similarly break many use-cases, hence why it’s not something we’re just throwing over the fence

44x4notfound hey all, I have cloudflare workers types in my tsconfig but it can't seem to fig...

Emilio•12/20/22, 5:18 PM

Here are the type definitions https://github.com/cloudflare/workerd/blob/main/types/defines/queues.d.ts -- hopefully they make it to a release soon

GitHub

workerd/queues.d.ts at main · cloudflare/workerd

The JavaScript / Wasm runtime that powers Cloudflare Workers - workerd/queues.d.ts at main · cloudflare/workerd

Pato•12/20/22, 5:48 PM

hey!

Pato•12/20/22, 5:48 PM

I was wondering. if I need to send a daily statement to a lot of users, can queues help me with this?

PPato I was wondering. if I need to send a daily statement to a lot of users, can queu...

John Spurlock•12/20/22, 5:51 PM

absolutely, insert the work with a daily croncron - either as separate messages or one large message, then make incremental progress inside the queuequeue handler

Pato•12/20/22, 5:53 PM

And by progress inside the queue handler you mean using a ConsumerConsumer?

PPato And by progress inside the queue handler you mean using a `Consumer`?

John Spurlock•12/20/22, 5:54 PM

yes, once you designate a worker as the queue consumer, it will call your queuequeue handler for any "work items" (messages of reports to send) https://developers.cloudflare.com/queues/javascript-apis/#consumer

Pato•12/20/22, 5:54 PM

So you would recommend to have 2 separate worker right?

John Spurlock•12/20/22, 5:54 PM

keep in mind the cpu limits and other limits still apply, so you'll need to break it up into smaller items

Pato•12/20/22, 5:55 PM

one that adds to the queue and one that only reads from it?

John Spurlock•12/20/22, 5:55 PM

personally I just have one, a worker can produce items and consume them

Pato•12/20/22, 5:55 PM

I see

John Spurlock•12/20/22, 5:55 PM

no need to overcomplicate unless the codebases are really different (and require different scripts)

Pato•12/20/22, 5:56 PM

What would be the best practice for this use case:
- My app produces daily statements (for some clients, not necessarily all of them)
- I need to fetch those statements from the accounts that have them and send them by email

John Spurlock•12/20/22, 6:01 PM

without knowing much about your setup, at a high level: have croncron calculate and materialize the work for the day, publish at least one queue msg when done, then handle/send those reports in the queuequeue handler. You'll need to decide whether to keep all the state in the messages themselves, or use storage (like in a DO) - so you don't send the same email twice (queues can sometimes send you the same message twice)

John Spurlock•12/20/22, 6:03 PM

if it were me, i'd go with the latter: keep the work-items in a DO, state for clientid+reportday+emailstatus, then just have messages that keep making progress on that list till it's complete

John Spurlock•12/20/22, 6:07 PM

could use R2 for the state too, or if you have an existing system db external to CF that would work too

kian•12/20/22, 6:18 PM

changes and lastRowId are unimplemented - at least last I saw on Nov 9th

Is there a way to see the queue length?

Similar Threads