Cloudflare Developers•2y ago

https://developers.cloudflare.com/queues/configuration/batching-retries/#batching

Cloudflare Docs

Batching, Retries and Delays · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock https://developers.cloudflare.com/queues/configuration/batching-retries/#batchin...

Deep•5/10/24, 4:12 AM

yup you are right, but my queue stops sending messages after the first batch id delivered and acknowledged

Dawson•5/11/24, 1:52 PM

Would this be a good use case for stripe billing? I need to charge the user per request

Dawson•5/11/24, 1:52 PM

I will pretty easily exceed the 100 operations a second rate limiting that Stripe applies. I could also scan through the db every 10 mins or so and bill them that way

johtso•5/11/24, 8:41 PM

if I disable concurrency, is there still a chance of a message being delivered more than once?

johtso•5/11/24, 8:43 PM

I have a usecase where I'm interacting with an external service that makes it very hard to achieve idempotency (it's only possible to check if an email has already been sent after waiting about a minute for it to be "indexed")

johtso•5/11/24, 8:44 PM

so if there's an error I need to make sure not to reattempt the job until a certain amount of time has elapsed

johtso•5/11/24, 8:44 PM

otherwise I risk sending users duplicate emails

LordSilver•5/12/24, 9:18 AM

you have no info from the dashboard that a queue is a DLQ of another queue?

LordSilver•5/12/24, 9:22 AM

how can you see event notification in your dashboard? https://developers.cloudflare.com/r2/buckets/event-notifications/

Cloudflare Docs

Event notifications · Cloudflare R2 docs

Event notifications send messages to your queue when data in your R2 bucket changes. You can consume these messages with a consumer Worker or pull …

LordSilver•5/12/24, 9:22 AM

after you created it

LordSilver•5/12/24, 9:23 AM

I went to check on the R2 bucket and also on the queue, but cannot see which triggers are configured

LordSilver•5/12/24, 9:23 AM

or maybe this is completely missing from the dashboard at the moment?

Deep•5/13/24, 5:20 PM

I have a question, what happens when one batch is procesing by the consumer worker, and the next batch is also full in the queue, and we are still adding the messages in queue?

Unsmart•5/13/24, 5:52 PM

Im a bit confused by the question, the messages would still be added to the queue and get processed eventually

LordSilver•5/14/24, 2:32 PM

what's the command to list all event notifications for a given r2 bucket?

UUnsmart Im a bit confused by the question, the messages would still be added to the queu...

Deep•5/15/24, 2:58 AM

so if the comsumer workers are not free where do the queue send the next batch to?

DDeep so if the comsumer workers are not free where do the queue send the next batch t...

Unsmart•5/15/24, 3:34 AM

No where it stays in the queue. Think of queues like waiting in line for a cashier... if a cashier is open you can go to checkout, if not you just wait until one is open

UUnsmart No where it stays in the queue. Think of queues like waiting in line for a cashi...

Deep•5/15/24, 4:18 PM

thank you for clearing , so what i understand is
"a queue can have a large number of messages in waiting state in backlog,
and when the consumers are free it sends BATCH_SIZE number of messages in bulk everytime"

so just one more question i am using browser rendering in the crawler and it has a limit of 2 new browsers per minute per account, so if that limit gives an error should I add retry delay in the queue? and if I give msg.retry() for one message from a batch, will the queue send only this message or send another batch including this message?

Deep•5/15/24, 4:37 PM

Also I launch a browser session with { keep_alive : 600000 }, so it should stay alive for 10 minutes, I am not doing browser.close() too, instead i am doing browser.disconnect(), then why it is getting released after 60 seconds of inactivity

Paul•5/15/24, 9:56 PM

So my understanding is that a queue can only have a single consumer. How would we handle a 1 to many situation? Let’s say I fire an event when a user signs up, so I place a “UserSignedUp” event on the queue. Now I’d like to have a bunch of consumers that are decoupled and can react to that event (notifying, saving to database, etc). They all do incredibly important operations, so I need a guaranteed delivery to each consumer. Is this possible in Cloudflare Queues?

Paul•5/15/24, 10:26 PM

I’m also seeing Pub/Sub is offered but without >= “at least once” guarantees.

PPaul So my understanding is that a queue can only have a single consumer. How would w...

Matt SilverlockOP•5/16/24, 12:00 AM

Queues guarantees delivery, but with at least once, so in some cases a message may be delivered more than once: https://developers.cloudflare.com/queues/reference/delivery-guarantees/

Cloudflare Docs

Delivery guarantees · Cloudflare Queues

Delivery guarantees define how strongly a messaging system enforces the delivery of messages it processes.

MMatt Silverlock Queues guarantees delivery, but with at least once, so in some cases a message m...

Paul•5/16/24, 12:01 AM

But with most other queue based systems, there can be many consumers per queue..

Paul•5/16/24, 12:01 AM

Which enables a nice decoupled event driven system

PPaul But with most other queue based systems, there can be many consumers per queue..

Matt SilverlockOP•5/16/24, 12:02 AM

Yes, but how are you decoupling each consumer here? Multiple consumers are still going to receive the same events — what it sounds like you want it a way to batch based on a key (defining an event type)

Paul•5/16/24, 12:02 AM

The only other way I can think to do it is have the main consumer as a “dispatch” and have it send the event back out on a bunch of other queues, but this sounds terrible

MMatt Silverlock Yes, but how are you decoupling each consumer here? Multiple consumers are still...

Paul•5/16/24, 12:03 AM

I’m looking for essentially pub/sub but with at least once guarantee

PPaul I’m looking for essentially pub/sub but with at least once guarantee

Matt SilverlockOP•5/16/24, 12:05 AM

Can you describe more here? A Pub/Sub consumer is going to pull something off the topic, but it’s all the same set of messages. I don’t see how this is any more decoupled: a single “UserSignedUp” event is still going to a single consumer.

Matt SilverlockOP•5/16/24, 12:05 AM

Publisher writes UserSignedUp event to /sign-ups topic -> consumer pulls off the topic (messages do not persist!) -> now consumer has to fire off all of those events (write to DB, send email, etc)

MMatt Silverlock Publisher writes UserSignedUp event to /sign-ups topic -> consumer pulls off the...

Paul•5/16/24, 12:08 AM

I would like a consumer dedicated to writing to DB, and a separate consumer dedicated to notification, etc.

I believe these are called “consumer groups”, like in Kafka.

Now in the future, if I want to perform some other completely decoupled operation, I can simply add another consumer group to the queue (and I don’t have to modify any of the existing consumer processes)

Unsmart•5/16/24, 12:08 AM

I think the easiest product to compare to is SNS.

Basically SNS -> (many different) SQS -> Lambda but on cloudflare.

Paul•5/16/24, 12:10 AM

Then there’s the whole replay feature, in case new consumer groups need to replay the events from some set point — but then you are just reimplementing Kafka , but damn that would be insanely powerful

PPaul I would like a consumer dedicated to writing to DB, and a separate consumer dedi...

Matt SilverlockOP•5/16/24, 12:12 AM

Got it, which means you need some way of fanning out and consuming the same event/message N-many times for (actively!) connected subscribers.

Something we’ve discussed but no timeline on that yet.

MMatt Silverlock Got it, which means you need some way of fanning out and consuming the same even...

Paul•5/16/24, 12:12 AM

Yes, and have that “at least once” guarantee for each of the subscribers

Paul•5/16/24, 12:13 AM

…since there could be multiple critical parallel operations I’m running on that event

johtso•5/16/24, 1:36 PM

what's the best practice for type safe queues? define a zod schema and check the payload, and reject it if it doesn't match the expected shape?

johtso•5/16/24, 1:37 PM

also is there not a way to expicitly reject a message and have it go directly to the DLQ?

Jjohtso what's the best practice for type safe queues? define a zod schema and check the...

Hello, I’m Allie!•5/16/24, 1:41 PM

Assuming you control all of the producers, I would probably not bother validating it on the consumer, since consumers should only ever push valid messages

Jjohtso also is there not a way to expicitly reject a message and have it go directly to...

Hello, I’m Allie!•5/16/24, 1:42 PM

Ack the message then push to the DLQ manually? It isn't quite the same, but it performs the same task

HHello, I’m Allie!Ack the message then push to the DLQ manually? It isn't quite the same, but it p...

johtso•5/16/24, 2:23 PM

yeah, thought that was probably the way to go

HHello, I’m Allie!Assuming you control all of the producers, I would probably not bother validatin...

johtso•5/16/24, 2:23 PM

sure, the message might have been valid once upon a time, but code changes

johtso•5/16/24, 2:24 PM

a key being renamed, and the consumer still expecting it to be there could mean an

undefined

undefined

causing all sorts of problems

Jjohtso sure, the message might have been valid once upon a time, but code changes 🙂

Hello, I’m Allie!•5/16/24, 2:26 PM

Maybe version it? Add a key at the top level with a version ID, then check just that

johtso•5/16/24, 2:31 PM

when you have Zod, everything looks like a nail

Jjohtso when you have Zod, everything looks like a nail 😂

Hello, I’m Allie!•5/16/24, 2:33 PM

Yeah, Zod is definitely the one-hit hammer here. Just think it is a little heavy-handed to validate the entire thing when you control both sides anyway

Hello, I’m Allie!•5/16/24, 2:33 PM

But IDK, both work

Michael Wittig•5/16/24, 3:23 PM

I started to experiment with R2 Event Notifications over Queues using http pull. My feedback:
1) Default Api limit of 1200 reqs/5min is 4 req/sec seems way to low. I have 100+ EC2 instances working on a queue. I just requested a limit increase to 30000 reqs/5min. I hope it will be approved. I never hit an SQS rate limit...
2) One of the most important missing features is to extend the visibility timeout while working on a message. At the moment I can only set the visibility timeout when I poll a message. At this time, I might not know how long I need to process it.
3) Jurisdiction support is missing.
4) The R2 event should contain a property jurisdiction.
I hope that helps.

https://developers.cloudflare.com/queues/configuration/batching-retries/#batching

Similar Threads