Cloudflare Developers•2y ago

when will queue remove beta and ready for production

AAlvee when will queue remove beta and ready for production

Matt Silverlock•2/10/24, 12:06 PM

queues

Rrenzor Did http pull ever land?

John Spurlock•2/11/24, 3:36 PM

queues

MMatt Silverlock Plan is this month 🙂

switz•2/13/24, 9:38 PM

just wondering, does this also mean that queues can hold items for consumption later? if so, how long will they remain in the queue?

switz•2/13/24, 9:39 PM

not just delay, I mean if the http pull isn't available and doesn't come online for 1h-24h

Sswitz not just delay, I mean if the http pull isn't available and doesn't come online ...

Matt Silverlock•2/13/24, 9:41 PM

If you don’t consume from a Pull consumer they will stay in the queue for the retention limit - 96 hours: https://developers.cloudflare.com/queues/platform/limits/

Cloudflare Docs

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

switz•2/13/24, 9:42 PM

fantastic, thank you for the info

switz•2/13/24, 9:43 PM

just to drop a link as you guys work on http pull, want to share some prior art in sqs-consumer which I find to be a very helpful API https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer

switz•2/13/24, 9:44 PM

unclear if you'll release a client library alongside http pull or its just an api, but I like using this with sqs

dattaproffs•2/14/24, 8:32 PM

hi, is it bad practice to have a worker with both a

fetch()

fetch()

and a queue()queue() ? the fetch is a simple web page. I'm have a litte troubble understanding what happens when the queuequeue forces the worker to scale? Is there any potential problems there?

Ddattaproffs hi, is it bad practice to have a worker with both a `fetch()` and a `queue()` ? ...

Hello, I’m Allie!•2/14/24, 10:15 PM

Nope, not at all. You can have every handler in a single Worker if you want

HHello, I’m Allie!Nope, not at all. You can have every handler in a single Worker if you want

dattaproffs•2/14/24, 10:17 PM

That's what i thought, but couldn't find any direct confirmation. anywhere in docs or so i can read about it? Thank's for the answer

Yigal•2/15/24, 6:28 AM

I have a situation where I have batches writing to KV, but if there are more than one update to the same key in one batch, the KV is likely to be overwritten and miss updates.
I thought it might be useful to hold the operation of the next batch\message until the first one ends.
How would you go about this kind of issue?

Yigal•2/15/24, 5:04 PM

Is there a way to control the time gap between each batch?

YYigal Is there a way to control the time gap between each batch?

Matt Silverlock•2/15/24, 11:33 PM

You can set the batch interval. If you need to delay messages, then see queues

piffie•2/22/24, 2:31 PM

Are there statistics available how long it takes for a queue to send it to an consumer? Is it typically less than a second? (not counting retries or something, just the typical case)

AApro Can we implement a queue consumer that is not a worker?

Matt Silverlock•2/27/24, 5:52 AM

Soon: queues

Matt Silverlock•2/27/24, 5:53 AM

We're also going to be changing the default content type — this will NOT be a breaking change as this will only impact new Workers and/or users that use the compatibility flag — to

json

json

, from the existing v8v8 default. This will improve compatibility with pull-based consumers when we release them.

MMatt Silverlock We're also going to be changing the default content type — this will NOT be a br...

Matt Silverlock•2/27/24, 5:54 AM

Docs PR: https://github.com/cloudflare/cloudflare-docs/pull/13170 - the target date for the new default is 2024-03-042024-03-04

GitHub

queues: Update javascript-apis.md by elithrar · Pull Request #13170...

Note: Merge this first - #13162
This PR:

Adds a changelog entry for the content type default change: #13162
Updates the QueuesContentType docs
Updates the How Queues Work doc to highlight this ...

mzadz•2/27/24, 8:02 PM

Hey question - if you place your consumer for a queue in the same worker that calls it, does it spawn a separate "process" or w/e the right terminology is, that is has its own resources? Or does it run in the same worker process?

Conner•2/27/24, 8:32 PM

Question on batch splitting if anyone knows answer:

If my batch messages are like { task_id, subtask_data }{ task_id, subtask_data }, and when I call sendBatch all messages in the batch have the same task_idtask_id, then can I guarantee that all messages in the consumer have the same task_idtask_id? Or will retries or max batch sizes mess with this?

Mmzadz Hey question - if you place your consumer for a queue in the same worker that ca...

Sam•2/27/24, 9:50 PM

Spawns a new worker

CConner Question on batch splitting if anyone knows answer: If my batch messages are li...

Matt Silverlock•2/28/24, 8:16 AM

No, you can't guarantee that today. The batches you write can't be the same as the batches you receive - because you might write smaller batches than you want to consume (max_batch_size) or wait for (max_wait_time).

Alisson Acioli•2/28/24, 7:48 PM

Hello, is there any estimate on when the feature of adding a delay will be released if a retry occurs in a message? My messages are trying immediately after failing and then leaving the queue, causing me to lose data

Sébastien Morel•2/28/24, 8:33 PM

hello here, I could debug that sending a simple payload to a queue is making my producer worker crazy slower. would you see a reason? Do queues have locations ?

Sébastien Morel•2/28/24, 8:40 PM

I am talking about hundred of milliseconds to .send.send to a queue. (that if I understood it right) must be awaited...
I can't be the only one experiencing that, right ?

Sébastien Morel•2/28/24, 8:51 PM

ha! that's interesting

Sébastien Morel•2/28/24, 8:51 PM

cause I have logged more, and the sendsend takes between 200ms and 300ms

Sébastien Morel•2/28/24, 8:53 PM

I think the document should mention that, cause I won't be the last following it

Sébastien Morel•2/28/24, 8:54 PM

https://developers.cloudflare.com/queues/reference/how-queues-works/#producers

Sébastien Morel•2/28/24, 9:04 PM

thank you @nora once again!

AAlisson Acioli Hello, is there any estimate on when the feature of adding a delay will be relea...

dsds10•2/28/24, 9:15 PM

Maybe you could try disabling retry and instead use a dead letter queue with higher timeout, or store fail state in kv and check every message if it was retried, then just let it wait the amount of time you need (since you don‘t pay for wall clock time anyway)

dsds10•2/28/24, 9:24 PM

What I‘m wondering (and sorry if it was already answered here, couldn‘t find it) is, why is there a generous limit of 10k queues per account when you can‘t produce/consume messages outside of a worker? Is this a prep for the http pull? But then again, you can only produce messages with a queue binding, which won‘t let you create a queue per user (at least outside of enterprise plan) although the changelog says this limit is „enabling easier per-user […] use-cases“

MMatt Silverlock No, you can't guarantee that today. The batches you write can't be the same as t...

Conner•2/29/24, 12:39 AM

Smaller batches is okay. I'm worried if fragmented batches might be mixed back together? Sounds like yes

CConner Smaller batches is okay. I'm worried if fragmented batches might be mixed back t...

Matt Silverlock•2/29/24, 12:41 AM

Correct - batches aren’t kept separate because most folks don’t want that. If you do, then set your max batch size to 1 + send your batches as a single array when calling send()

CConner Smaller batches is okay. I'm worried if fragmented batches might be mixed back t...

Matt Silverlock•2/29/24, 12:42 AM

We’ve discussed allowing batching by an object/JSON key, which would achieve this correctly. No timeline yet.

Conner•2/29/24, 12:44 AM

Makes sense okay

Conner•2/29/24, 12:44 AM

Yeah the option for consumer delay / send delay is a bigger ask haha
Would love to see that

MMatt Silverlock We’ve discussed allowing batching by an object/JSON key, which would achieve thi...

Conner•2/29/24, 12:46 AM

Extra note on this though
For batching by parent key, would be amazing if another event was fired for when all child tasks all done
So could track success/fail of parent based on child tasks
Having to hit db everytime for that rn

Alisson Acioli•3/1/24, 11:48 AM

@Matt Is there a forecast for the delay in messages that failed to be rerun again?

Alex•3/1/24, 9:02 PM

Has anyone been having very long delays for messages to be picked up by consumers?

Matt Silverlock•3/4/24, 1:02 PM

FYI - this is now live: https://developers.cloudflare.com/queues/platform/changelog/#default-content-type-now-set-to-json

Cloudflare Docs

Changelog · Cloudflare Queues

The default content type for messages published to a queue is now json, which improves compatibility with the upcoming pull-based queues.

Uulrikstrid Is there any documentation for running queues locally, or at least using hosted ...

Vero•3/6/24, 3:10 PM

https://developers.cloudflare.com/queues/reference/local-development/

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

when will queue remove beta and ready for production

Similar Threads

Similar Threads

Similar Threads