Yeah same, basically the message ids `A`, `B`, etc will relate to an item in the database which I wa

Yeah same, basically the message ids

, etc will relate to an item in the database which I want to operate on in some way with each message in the queue. I dont really care about the FIFO nature for that just that messages are grouped so I only do operations to a single message group id and not hope for the best

UnsmartOP•3/1/23, 10:31 PM

Thats what explicit acknowledgement should solve afaik

kian•3/1/23, 10:31 PM

UUnsmart Thats what explicit acknowledgement should solve afaik

Matt Silverlock•3/1/23, 10:36 PM

Explicit acknowledgement/negative acknowledgement yes. No backoff/timeout (yet)

UUnsmart Yeah same, basically the message ids `A`, `B`, etc will relate to an item in the...

Matt Silverlock•3/1/23, 10:38 PM

Got it - both of you effectively want partitioning within a queue - e.g. a

batchKey

batchKey

that only delivers a batch of messages with the same (arbitrary) key.

MMatt Silverlock Got it - both of you effectively want partitioning within a queue - e.g. a `batc...

UnsmartOP•3/1/23, 10:39 PM

yeah exactly, crazy amount of ids, fairly low throughput

UnsmartOP•3/1/23, 10:41 PM

I probably wouldnt even need queues at all if I used durable objects tbh but dynamodb transactions are very rough so I have to use a queue

Matt Silverlock•3/1/23, 10:43 PM

Dynamic bindings/Queues == roughly the same as arbitrary batch keys for your use-case. Probably easier (no need to wire up a consumer each time).

Matt Silverlock•3/2/23, 5:38 PM

The per-queue throughput limit has now been raised to 400 msg/sec per queue (up from 100 msg/sec). This limit will continue to increase over the course of the beta. https://developers.cloudflare.com/queues/changelog#higher-per-queue-throughput

MMatt Silverlock 📣 The per-queue throughput limit has now been raised to 400 msg/sec per queue ...

Vero•3/2/23, 6:31 PM

@Queues

Matt Silverlock•3/2/23, 6:42 PM

New feature - Queues now supports explicit acknowledgement, allowing you to

.ack()

.ack()

individual messages as you process them, and prevent cases where batches need to be re-delivered in full. You can also force a batch to be retried, or even ack an entire batch preemptively. Read more here: https://developers.cloudflare.com/queues/changelog/#explicit-acknowledgement-new-feature

NNeves I also heard about a way to retry only the failed ones in the batch. Is there a ...

Matt Silverlock•3/2/23, 8:29 PM

https://developers.cloudflare.com/queues/changelog/#explicit-acknowledgement-new-feature

Changelog · Cloudflare Queues

You can now acknowledge individual messages with a batch by calling .ack() on a message.

FFriday Candour I still don't know what I can use queues for even am very excited about every cl...

Matt Silverlock•3/2/23, 9:06 PM

https://blog.cloudflare.com/cloudflare-queues-open-beta/ + https://developers.cloudflare.com/queues/

Jadugar_Jaggu•3/3/23, 11:02 AM

Hello I tried Queue, I found limitations on message batch size and wait time which is not good I think cause for high traffic apps 100 batch message size can fill in a fraction of seconds

Jadugar_Jaggu•3/3/23, 11:02 AM

Then whats the point of calling it queue ?

Jadugar_Jaggu•3/3/23, 11:11 AM

There are two very big limitations that should not be in the queue

First is limitations on message batch size to 100 messages only and message wait time to 30 seconds only max

second is if 100 messages are filled within seconds it sends it to the consumer worker right away which is I think same as hitting my API every second
There should be a delay system for consumer workers otherwise it is nt good, we can make a better queue using Cloudflare KV and Cron triggers

UnsmartOP•3/3/23, 2:54 PM

Queues allow you to process messages in the background when you don't need it immediately so if there's a backlog of messages because of high traffic it won't be an issue. Eventually all the messages will be processed.

Queues are in beta limits are subject to change once they work things out.

Eventually you will be able to use HTTP to poll queues if you wanted but I doubt queues will have any delay system like that.

Wwarpandas What's the best way to extract the `body` values from a `MessageBatch` object in...

Matt Silverlock•3/4/23, 1:11 PM

Loop over the batch and push each

msg.body

msg.body

into an array? What have you tried?

Matt Silverlock•3/4/23, 1:13 PM

let bodies = []
for (let msg of batch.messages) {
   bodies.push(msg.body)
}

let bodies = []
for (let msg of batch.messages) {
   bodies.push(msg.body)
}

… is the minimal example. Also: what are you trying to do beyond that?

$\ ឵឵឵$ \\ ឵឵឵Minimum would be adding a `delay` param to queue config, which would allow build...

Matt Silverlock•3/4/23, 7:50 PM

Definitely something we’re considering.

Likely on both

.send()

.send()

and

.retry()

.retry()

- as an example and indicative only:

•

.send(msg, { delay: 600 })

.send(msg, { delay: 600 })

•

.retry({ delay: 300, backoffFactor: 2.0, maxDelay: 3600 })

.retry({ delay: 300, backoffFactor: 2.0, maxDelay: 3600 })

However - there are some sharp edges to work through around getting the existing delay config of a message, and avoiding cases where misconfiguration causes you to delay messages more than expected (e.g. hours instead of minutes). Important for us to help developers avoid putting themselves in a state they can’t get out of.

UnsmartOP•3/6/23, 7:43 PM

github taking like 6 years to load that but

UnsmartOP•3/6/23, 7:44 PM

I wonder what the max concurrency you can set will be

UUnsmart I wonder what the max concurrency you can set will be

Matt Silverlock•3/6/23, 9:18 PM

Leave it enabled and let us keep growing it

Matt Silverlock•3/6/23, 9:19 PM

Goal is to just scale as much as needed, with a control for maxConcurrency so folks can limit upstream tput where necessary.

UnsmartOP•3/7/23, 1:17 AM

Oh nice

Jacob•3/8/23, 11:47 PM

What happens to a currently running batch when you publish a new version of a consumer Worker? Does it finish processing that batch on the old version?

JJacob What happens to a currently running batch when you publish a new version of a co...

Matt Silverlock•3/9/23, 1:13 AM

Yes, any running invocations run to completion. New invocations will use the new script version.

Matt Silverlock•3/9/23, 1:13 AM

Same for any Worker

Jacob•3/9/23, 1:14 AM

Perfect, thanks! Just double checking

Was deploying while it was working through a 2.7k backlog and wanted to make sure I wasn't breaking it

Jacob•3/9/23, 1:15 AM

It's so fun watching it go pogPepe

Matt Silverlock•3/9/23, 1:17 AM

If you want it to go faster you’ll be able to let it scale our horizontally soon too - or just limit it to 1

Jacob•3/9/23, 1:19 AM

That's awesome! Sadly this particular queue is just waiting on Discord rate limits, but I'm super excited about trying more scalable workloads on Queues

Jacob•3/9/23, 1:20 AM

One thing I'd like to try with Queues that would benifit from horizontal scaling: I have about 3 million files in R2 that I accidently saved under the wrong paths. It would be cool to use Queues to move them all

Jacob•3/9/23, 1:23 AM

each one would need to read the file, parse out a timestamp, and then move it

MMatt Silverlock If you want it to go faster you’ll be able to let it scale our horizontally soon...

Jacob•3/9/23, 1:24 AM

When will that be out? I'd love to try it this weekend

JJacob When will that be out? I'd love to try it this weekend 🙂

Matt Silverlock•3/9/23, 1:25 AM

Probably next week?

- Docs PR: https://github.com/cloudflare/cloudflare-docs/pull/7932
- WIP wrangler PR you can probably check out or grab the API endpoints from: https://github.com/cloudflare/workers-sdk/pull/2859

GitHub

queues: consumer concurrency by elithrar · Pull Request #7932 · clo...

New feature: consumer concurrency.

Changelog
Limits page updated
Update /platform/configuration/ with new properties
wrangler updates merged from cloudflare/workers-sdk#2859
New /learning/con...

GitHub

Add support for Queue Consumer concurrency by jbwcloudflare · Pull ...

What this PR solves / how to test:
This PR adds support for new Queues Consumer configuration options: concurrency_enabled and max_concurrency.
Associated docs issues/PR: In progress
Author has inc...

Yeah same, basically the message ids `A`, `B`, etc will relate to an item in the database which I wa

Similar Threads

Similar Threads

Similar Threads