Question on batch splitting if anyone knows answer: If my batch messages are like `{ task_id, subta

Question on batch splitting if anyone knows answer:

If my batch messages are like

{ task_id, subtask_data }

{ task_id, subtask_data }

{ task_id, subtask_data }

{ task_id, subtask_data }, and when I call sendBatch all messages in the batch have the same

task_id

task_id

task_id

task_id, then can I guarantee that all messages in the consumer have the same

task_id

task_id

task_id

task_id? Or will retries or max batch sizes mess with this?

Mmzadz Hey question - if you place your consumer for a queue in the same worker that ca...

Sam•2/27/24, 9:50 PM

Spawns a new worker

CConner Question on batch splitting if anyone knows answer: If my batch messages are li...

Matt Silverlock•2/28/24, 8:16 AM

No, you can't guarantee that today. The batches you write can't be the same as the batches you receive - because you might write smaller batches than you want to consume (max_batch_size) or wait for (max_wait_time).

Alisson Acioli•2/28/24, 7:48 PM

Hello, is there any estimate on when the feature of adding a delay will be released if a retry occurs in a message? My messages are trying immediately after failing and then leaving the queue, causing me to lose data

Sébastien Morel•2/28/24, 8:33 PM

hello here, I could debug that sending a simple payload to a queue is making my producer worker crazy slower. would you see a reason? Do queues have locations ?

Sébastien Morel•2/28/24, 8:40 PM

I am talking about hundred of milliseconds to .send.send to a queue. (that if I understood it right) must be awaited...
I can't be the only one experiencing that, right ?

Sébastien Morel•2/28/24, 8:51 PM

ha! that's interesting

Sébastien Morel•2/28/24, 8:51 PM

cause I have logged more, and the sendsend takes between 200ms and 300ms

Sébastien Morel•2/28/24, 8:53 PM

I think the document should mention that, cause I won't be the last following it

Sébastien Morel•2/28/24, 8:54 PM

https://developers.cloudflare.com/queues/reference/how-queues-works/#producers

Sébastien Morel•2/28/24, 9:04 PM

thank you @nora once again!

AAlisson Acioli Hello, is there any estimate on when the feature of adding a delay will be relea...

dsds10•2/28/24, 9:15 PM

Maybe you could try disabling retry and instead use a dead letter queue with higher timeout, or store fail state in kv and check every message if it was retried, then just let it wait the amount of time you need (since you don‘t pay for wall clock time anyway)

dsds10•2/28/24, 9:24 PM

What I‘m wondering (and sorry if it was already answered here, couldn‘t find it) is, why is there a generous limit of 10k queues per account when you can‘t produce/consume messages outside of a worker? Is this a prep for the http pull? But then again, you can only produce messages with a queue binding, which won‘t let you create a queue per user (at least outside of enterprise plan) although the changelog says this limit is „enabling easier per-user […] use-cases“

MMatt Silverlock No, you can't guarantee that today. The batches you write can't be the same as t...

ConnerOP•2/29/24, 12:39 AM

Smaller batches is okay. I'm worried if fragmented batches might be mixed back together? Sounds like yes

CConner Smaller batches is okay. I'm worried if fragmented batches might be mixed back t...

Matt Silverlock•2/29/24, 12:41 AM

Correct - batches aren’t kept separate because most folks don’t want that. If you do, then set your max batch size to 1 + send your batches as a single array when calling send()

CConner Smaller batches is okay. I'm worried if fragmented batches might be mixed back t...

Matt Silverlock•2/29/24, 12:42 AM

We’ve discussed allowing batching by an object/JSON key, which would achieve this correctly. No timeline yet.

ConnerOP•2/29/24, 12:44 AM

Makes sense okay

ConnerOP•2/29/24, 12:44 AM

Yeah the option for consumer delay / send delay is a bigger ask haha
Would love to see that

MMatt Silverlock We’ve discussed allowing batching by an object/JSON key, which would achieve thi...

ConnerOP•2/29/24, 12:46 AM

Extra note on this though
For batching by parent key, would be amazing if another event was fired for when all child tasks all done
So could track success/fail of parent based on child tasks
Having to hit db everytime for that rn

Alisson Acioli•3/1/24, 11:48 AM

@Matt Is there a forecast for the delay in messages that failed to be rerun again?

Alex•3/1/24, 9:02 PM

Has anyone been having very long delays for messages to be picked up by consumers?

Matt Silverlock•3/4/24, 1:02 PM

FYI - this is now live: https://developers.cloudflare.com/queues/platform/changelog/#default-content-type-now-set-to-json

Cloudflare Docs

Changelog · Cloudflare Queues

The default content type for messages published to a queue is now json, which improves compatibility with the upcoming pull-based queues.

Uulrikstrid Is there any documentation for running queues locally, or at least using hosted ...

Vero•3/6/24, 3:10 PM

https://developers.cloudflare.com/queues/reference/local-development/

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

alex•3/6/24, 6:52 PM

Is there some library for Go?

Sharkey•3/7/24, 12:28 AM

How has the general reliability of the beta been?

We are thinking of using CF queues to build our notification system off of in production. Is it production ready?

Prakash•3/7/24, 2:44 AM

I am facing issues while working on Queues locally, please help.

Prakash•3/7/24, 2:44 AM

Error

platform:dev: ✘ [ERROR] Uncaught (async) Error: Handler does not export a queue() function.
platform:dev: 
[wrangler-UserWorker:wrn] Dropped message "e1d974b8e338ba4bd883f45d1f630728" on queue "queue" after 3 failed attempts!

platform:dev: ✘ [ERROR] Uncaught (async) Error: Handler does not export a queue() function.
platform:dev: 
[wrangler-UserWorker:wrn] Dropped message "e1d974b8e338ba4bd883f45d1f630728" on queue "queue" after 3 failed attempts!

Uulrikstrid Can I run queue's locally when developing? From what I can tell I can't put thin...

Prakash•3/7/24, 2:47 AM

Same here, did you find any solution?

John Spurlock•3/9/24, 11:07 PM

excited to start adding queues to production workers once it hits GA - two things I'm curious about:

1. does adding a QueueQueue environment binding limit the colos where the producing

fetch

fetch

handler can run, or change the behavior of the worker in any way?
2. does adding a queuequeue handler (consumer) in the same worker as an existing

fetch

fetch

handler limit where the

fetch

fetch

handler can run, or change the behavior of the fetch side of the worker in any way?

JJohn Spurlock excited to start adding queues to production workers once it hits GA - two thing...

Hello, I’m Allie!•3/10/24, 9:23 AM

1. No
2. No

HHello, I’m Allie!1. No 2. No

Matt Silverlock•3/10/24, 2:53 PM

Correct. The consumer (the queue() handler) is effectively its own entity. Not something you need to think about.

MMatt Silverlock Correct. The consumer (the queue() handler) is effectively its own entity. Not s...

John Spurlock•3/10/24, 2:53 PM

excellent, thanks!

DanGamble•3/10/24, 9:46 PM

If you retry a single message, does it go to the to the back of the queue or does it just get tried in the next batch?

Dalist•3/11/24, 9:17 AM

Hi everyone, is there any updates for this product yet?
https://developers.cloudflare.com/pub-sub/

Or is it built within Cloudflare Queues now?

Original message was deleted

Dalist•3/11/24, 9:20 AM

oh, didn't see that one, thank you @Leo

Uulrikstrid Is there a max time for how long time a consumer can take before it needs to be ...

John Spurlock•3/12/24, 4:15 PM

yes, it's 15 minutes

https://developers.cloudflare.com/queues/platform/limits/

switz•3/12/24, 7:34 PM

pull!

switz•3/12/24, 8:28 PM

question: with pull, will items sit in the queue for a long time if they are not pulled yet? whats the time limit?

zegevlier•3/12/24, 8:32 PM

Posting this here since it's relevant

https://fxtwitter.com/elithrar/status/1767629801500692709

FxTwitter / FixupX💬 4 🔁 1 ❤️ 23 👁️ 678

Matt Silverlock 🐀 (@elithrar)

Kicking the tires on a shiny new feature for @cloudflare Queues:

Sswitz question: with pull, will items sit in the queue for a long time if they are not...

zegevlier•3/12/24, 8:34 PM

Currently the limit is 4 days (see: https://developers.cloudflare.com/queues/platform/limits/)

Zzegevlier Posting this here since it's relevant 😄 https://fxtwitter.com/elithrar/status/...

switz•3/14/24, 5:35 PM

https://silverlock-queues-http-pull.cloudflare-docs-7ou.pages.dev/queues/reference/http-pull/

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

Jeroen•3/15/24, 2:43 AM

Is there no way to use cloudflare-queue without a worker (external service)? I am implementing an API that doesn't use HTTP, so I do not think I can use a worker for this.

Question on batch splitting if anyone knows answer: If my batch messages are like `{ task_id, subta

Similar Threads