Got it - both of you effectively want partitioning within a queue - e.g. a `batchKey` that only deli

Got it - both of you effectively want partitioning within a queue - e.g. a

batchKey

batchKey

batchKey

batchKey that only delivers a batch of messages with the same (arbitrary) key.

MMatt Silverlock Got it - both of you effectively want partitioning within a queue - e.g. a `batc...

Unsmart•3/1/23, 10:39 PM

yeah exactly, crazy amount of ids, fairly low throughput

Unsmart•3/1/23, 10:41 PM

I probably wouldnt even need queues at all if I used durable objects tbh but dynamodb transactions are very rough so I have to use a queue

Matt SilverlockOP•3/1/23, 10:43 PM

Dynamic bindings/Queues == roughly the same as arbitrary batch keys for your use-case. Probably easier (no need to wire up a consumer each time).

Matt SilverlockOP•3/2/23, 5:38 PM

The per-queue throughput limit has now been raised to 400 msg/sec per queue (up from 100 msg/sec). This limit will continue to increase over the course of the beta. https://developers.cloudflare.com/queues/changelog#higher-per-queue-throughput

MMatt Silverlock 📣 The per-queue throughput limit has now been raised to 400 msg/sec per queue ...

Vero•3/2/23, 6:31 PM

@Queues

Matt SilverlockOP•3/2/23, 6:42 PM

New feature - Queues now supports explicit acknowledgement, allowing you to .ack().ack() individual messages as you process them, and prevent cases where batches need to be re-delivered in full. You can also force a batch to be retried, or even ack an entire batch preemptively. Read more here: https://developers.cloudflare.com/queues/changelog/#explicit-acknowledgement-new-feature

NNeves I also heard about a way to retry only the failed ones in the batch. Is there a ...

Matt SilverlockOP•3/2/23, 8:29 PM

https://developers.cloudflare.com/queues/changelog/#explicit-acknowledgement-new-feature

Changelog · Cloudflare Queues

You can now acknowledge individual messages with a batch by calling .ack() on a message.

FFriday Candour I still don't know what I can use queues for even am very excited about every cl...

Matt SilverlockOP•3/2/23, 9:06 PM

https://blog.cloudflare.com/cloudflare-queues-open-beta/ + https://developers.cloudflare.com/queues/

Jadugar_Jaggu•3/3/23, 11:02 AM

Hello I tried Queue, I found limitations on message batch size and wait time which is not good I think cause for high traffic apps 100 batch message size can fill in a fraction of seconds

Jadugar_Jaggu•3/3/23, 11:02 AM

Then whats the point of calling it queue ?

Jadugar_Jaggu•3/3/23, 11:11 AM

There are two very big limitations that should not be in the queue

First is limitations on message batch size to 100 messages only and message wait time to 30 seconds only max

second is if 100 messages are filled within seconds it sends it to the consumer worker right away which is I think same as hitting my API every second
There should be a delay system for consumer workers otherwise it is nt good, we can make a better queue using Cloudflare KV and Cron triggers

Unsmart•3/3/23, 2:54 PM

Queues allow you to process messages in the background when you don't need it immediately so if there's a backlog of messages because of high traffic it won't be an issue. Eventually all the messages will be processed.

Queues are in beta limits are subject to change once they work things out.

Eventually you will be able to use HTTP to poll queues if you wanted but I doubt queues will have any delay system like that.

Wwarpandas What's the best way to extract the `body` values from a `MessageBatch` object in...

Matt SilverlockOP•3/4/23, 1:11 PM

Loop over the batch and push each msg.bodymsg.body into an array? What have you tried?

Matt SilverlockOP•3/4/23, 1:13 PM

let bodies = []
for (let msg of batch.messages) {
   bodies.push(msg.body)
}

let bodies = []
for (let msg of batch.messages) {
   bodies.push(msg.body)
}

… is the minimal example. Also: what are you trying to do beyond that?

$\ ឵឵឵$ \\ ឵឵឵Minimum would be adding a `delay` param to queue config, which would allow build...

Matt SilverlockOP•3/4/23, 7:50 PM

Definitely something we’re considering.

Likely on both .send().send() and .retry().retry() - as an example and indicative only:

• .send(msg, { delay: 600 }).send(msg, { delay: 600 })
• .retry({ delay: 300, backoffFactor: 2.0, maxDelay: 3600 }).retry({ delay: 300, backoffFactor: 2.0, maxDelay: 3600 })

However - there are some sharp edges to work through around getting the existing delay config of a message, and avoiding cases where misconfiguration causes you to delay messages more than expected (e.g. hours instead of minutes). Important for us to help developers avoid putting themselves in a state they can’t get out of.

Unsmart•3/6/23, 7:43 PM

github taking like 6 years to load that but

Unsmart•3/6/23, 7:44 PM

I wonder what the max concurrency you can set will be

UUnsmart I wonder what the max concurrency you can set will be

Matt SilverlockOP•3/6/23, 9:18 PM

Leave it enabled and let us keep growing it

Matt SilverlockOP•3/6/23, 9:19 PM

Goal is to just scale as much as needed, with a control for maxConcurrency so folks can limit upstream tput where necessary.

Unsmart•3/7/23, 1:17 AM

Oh nice

Jacob•3/8/23, 11:47 PM

What happens to a currently running batch when you publish a new version of a consumer Worker? Does it finish processing that batch on the old version?

JJacob What happens to a currently running batch when you publish a new version of a co...

Matt SilverlockOP•3/9/23, 1:13 AM

Yes, any running invocations run to completion. New invocations will use the new script version.

Matt SilverlockOP•3/9/23, 1:13 AM

Same for any Worker

Jacob•3/9/23, 1:14 AM

Perfect, thanks! Just double checking

Was deploying while it was working through a 2.7k backlog and wanted to make sure I wasn't breaking it

Jacob•3/9/23, 1:15 AM

It's so fun watching it go pogPepe

Matt SilverlockOP•3/9/23, 1:17 AM

If you want it to go faster you’ll be able to let it scale our horizontally soon too - or just limit it to 1

Jacob•3/9/23, 1:19 AM

That's awesome! Sadly this particular queue is just waiting on Discord rate limits, but I'm super excited about trying more scalable workloads on Queues

Jacob•3/9/23, 1:20 AM

One thing I'd like to try with Queues that would benifit from horizontal scaling: I have about 3 million files in R2 that I accidently saved under the wrong paths. It would be cool to use Queues to move them all

Jacob•3/9/23, 1:23 AM

each one would need to read the file, parse out a timestamp, and then move it

MMatt Silverlock If you want it to go faster you’ll be able to let it scale our horizontally soon...

Jacob•3/9/23, 1:24 AM

When will that be out? I'd love to try it this weekend

JJacob When will that be out? I'd love to try it this weekend 🙂

Matt SilverlockOP•3/9/23, 1:25 AM

Probably next week?

- Docs PR: https://github.com/cloudflare/cloudflare-docs/pull/7932
- WIP wrangler PR you can probably check out or grab the API endpoints from: https://github.com/cloudflare/workers-sdk/pull/2859

GitHub

queues: consumer concurrency by elithrar · Pull Request #7932 · clo...

New feature: consumer concurrency.

Changelog
Limits page updated
Update /platform/configuration/ with new properties
wrangler updates merged from cloudflare/workers-sdk#2859
New /learning/con...

GitHub

Add support for Queue Consumer concurrency by jbwcloudflare · Pull ...

What this PR solves / how to test:
This PR adds support for new Queues Consumer configuration options: concurrency_enabled and max_concurrency.
Associated docs issues/PR: In progress
Author has inc...

Jacob•3/9/23, 1:26 AM

Sweet! Looking forward to the release

Jacob•3/9/23, 1:26 AM

Btw can a Queue send to itself or is recursion not allowed?

Jacob•3/9/23, 1:26 AM

I guess you could always fetch to a Worker that sends back to it

Jacob•3/9/23, 1:27 AM

Trying to think of the best way to queue up 3 million items

JJacob Btw can a Queue send to itself or is recursion not allowed?

Matt SilverlockOP•3/9/23, 1:27 AM

You can do that if you want. We just don’t recommend it.

Matt SilverlockOP•3/9/23, 1:28 AM

R2 list() -> write keys to queue in sendBatch() -> paginate -> keep going. Store page state in a KV key or D1?

Jacob•3/9/23, 1:28 AM

Oh that could work. and then just keep hitting the endpoint until it pages through them all

Got it - both of you effectively want partitioning within a queue - e.g. a `batchKey` that only deli

Similar Threads

Similar Threads

Similar Threads