Hi team, I need to use event notifications with R2 and queues when new objects are created. I have a

Hi team, I need to use event notifications with R2 and queues when new objects are created. I have a job written in C#/.NET which is supposed to read messages from this queue. My question is: do you have a .NET SDK that I can use for receiving/acknowledging messages from queues? For example, I think R2 is compatible with AWS S3 SDK and I can use AmazonS3Client with R2, is that the case with queues? Can I use AmazonSqsClient with Cloudflare queues? Or what other options do I have?

Ccloudkite When you explicitly call message.retry() does this still respect the `max_retrie...

Pranshu Maheshwari•9/3/24, 2:34 PM

yes!

Llusine Hi team, I need to use event notifications with R2 and queues when new objects a...

Matt Silverlock•9/3/24, 11:33 PM

You can pull via HTTP - we don’t have a .NET SDK and the SQS API is far less of a defacto standard vs S3, and so we have no plans to support it. https://developers.cloudflare.com/queues/configuration/pull-consumers/

Cloudflare Docs

Pull consumers | Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A pull-based consumer can be useful when your message consumption rate is limited by upstream infrastructure or long-running tasks.

Ssouvikinator Hey guys I'm new to the Cloudflare ecosystem. Need some guidance. Here is my req...

Hard@Work•9/4/24, 9:17 AM

?crossposting

Flare•9/4/24, 9:17 AM

Please do not post your question in multiple channels/post it multiple times per the rules at ⁠#

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Ssouvikinator Hey guys I'm new to the Cloudflare ecosystem. Need some guidance. Here is my req...

harshil1712•9/4/24, 10:00 AM

I don't see why Queues won't be a good fit!

Ssouvikinator My concerns are batch processing, persistance and stability. I saw few posts abo...

harshil1712•9/4/24, 10:07 AM

I have built some demos with queues and have not faced the issues. One thing I have done to make sure that there is consistency is adding retries at the relevant places. You can also setup a dead-letter queue. Here's the doc that might be useful: https://developers.cloudflare.com/queues/configuration/batching-retries/#explicit-acknowledgement-and-retries

Cloudflare Docs

Batching, Retries and Delays | Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

Ssouvikinator great I need to try it out. So I assume it's ready for production right?

harshil1712•9/4/24, 10:12 AM

It's going to be GA soon

Ssouvikinator Hey also if you could explain me a bit because I'm still figuring out how the wo...

harshil1712•9/4/24, 10:53 AM

You can have the producer and the consumer in the same worker or in separate ones. It really depends on how you want to architect your application. However, for now, when building locally, we recommend having them in the same worker.
https://developers.cloudflare.com/queues/configuration/local-development/#known-issues

The pricing docs have the answer on the execution: https://developers.cloudflare.com/queues/platform/pricing/

Cloudflare Docs

Pricing | Cloudflare Queues

Cloudflare Queues charges for the total number of operations against each of your queues during a given month.

Cloudflare Docs

Local Development | Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as Cloudflare runs globally.

celer•9/4/24, 8:46 PM

Hi! Has anyone seen CPU Execution timeout on queue consumer event? In the worker limits docs they say the limit is 15 minutes of CPU time. But I always get timeout after 32,000ms (according to the worker cpu time dashboard). I could work around that, but I'd like to know if the docs are wrong or if it's a bug on my or CF side.

Michael Wittig•9/5/24, 12:49 PM

Hi! I deleted a bunch of R2 bucket notifications & buckets but I can not delete the queue that was referenced. I always receive the error "cannot delete queue bucketav-rhyglnin that serves as a target for event notifications". I have no buckets left so this is likely not the case. Are there any reqasons for that? Is there something happening in the background before I can delete the queue? If I look at the queue in the dashboard it says "This queue does not have a consumer". Clicking the delete button in the UI is broken as well. I can see the error in the browser logs but not the UI.

johtso•9/5/24, 2:42 PM

hmm, it seems like my messages are being retried immediately after an implicit retry instead of waiting the configured 60 seconds..

[[env.staging.queues.consumers]]
queue = "emails-staging"
max_batch_size = 1
max_concurrency = 1
dead_letter_queue = "emails-dlq-staging"
max_retries = 5
retry_delay = 60

[[env.staging.queues.consumers]]
queue = "emails-staging"
max_batch_size = 1
max_concurrency = 1
dead_letter_queue = "emails-dlq-staging"
max_retries = 5
retry_delay = 60

Stockton•9/7/24, 12:14 AM

What would be the benefits of having a producer and consumer in two different workers? Or vice versa?

micooz•9/8/24, 5:22 PM

Queue "consumer concurrency" feature does not seem to be suitable for scenarios that rely on sequential execution.

ytsruh•9/10/24, 1:59 PM

Hey all, does anybody if there is a way to get both Queues & the Browser Rendering API working nicely together in local dev? Seems Queues doesn't support --remote but Browser Rendering only works when using --remote?

Yytsruh Hey all, does anybody if there is a way to get both Queues & the Browser Renderi...

Sam•9/10/24, 2:56 PM

Did not have your problem, but you could try to just put your queue script inside the fetch() one to test it out remotely.

SSam Did not have your problem, but you could try to just put your queue script insid...

ytsruh•9/10/24, 3:02 PM

Thanks. I think we’re switching to scheduled instead. Our use case didn’t have massive throughput. Cheers for quick reply

OOle Melhus Hey! I'm seeing delays for up to 5-7 seconds between adding something to a queue...

harshil1712•9/11/24, 12:05 PM

Can you expan a bit more on this? Is the consumer queue not picking it up? Can you also share your wrangler.toml config for the queues?

psteinroe•9/12/24, 2:22 PM

hey! is there any (inofficial) goal for queues performance? will it ever be comparable to eg sqs? sending to the queue is currently multiiple 100ms, and it takes multiple seconds for a worker to receive it. and throughput is much much lower at the moment.

kcbrewron•9/13/24, 11:23 PM

I haven't run into messages dropping or batches dropping. I have built some interesting architecture where I have a message-producer API which abstracts all the applications from needing to know about how to send messages. We used a KV bound to the message producer API to know what queue to route to and bind the worker to any queue. I know there are limits to the binding, but this API has made it easy to onboard queues for a few different use cases. I'm ready for this to move to GA. What issues is the team waiting on solving before moving it to GA?

oof2win2•9/16/24, 9:40 AM

is there a plan to have messages be "grouped" and sent that way? i have a merchantId prop on my messages and i want to handle up to 20 messages per merchant, but there isn't really a way to do that right now

oof2win2•9/16/24, 9:41 AM

it would be really neat if i could do that honestly

VoidRanger•9/16/24, 10:15 PM

Hi there, just wanted to know if there is any way to see which buckets are sending notifications to a queue

tadhglewis•9/17/24, 1:07 PM

Couple queries:

Any plans to add a Message.deadletterMessage.deadletter method? I have some use cases where I know the message will always fail and it needs manual triage.

It'd be great to have redrive functionality to move messages from the deadletters to processing queue once any issues with the deadletters have been resolved (e.g. temp outage). AWS SQS has similar functionality https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html

Learn how to configure a dead-letter queue redrive in Amazon SQS - ...

Learn how to configure an Amazon SQS dead-letter queue redrive to move messages from a dead-letter queue to a source queue or other standard queue.

tadhglewis•9/17/24, 1:10 PM

Also, are there plans to suppport passing a

Message

Message

into an RPC Worker?

Jason Barry•9/18/24, 11:25 PM

I don't see wrangler.toml's equivalent of [[queues.consumers]][[queues.consumers]] referenced in https://developers.cloudflare.com/workers/configuration/multipart-upload-metadata/#bindings -- is that intentional? if so, how does one configure a Worker uploaded from the API to be a queue consumer? I've added "type": "queue""type": "queue" in my bindings metadata, but that only seems to register the worker as a queue producer

JJason Barry I don't see wrangler.toml's equivalent of `[[queues.consumers]]` referenced in h...

Hello, I’m Allie!•9/19/24, 6:33 AM

It's done via this API: https://developers.cloudflare.com/api/operations/queue-v2-create-queue-consumer

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

HHello, I’m Allie!It's done via this API: https://developers.cloudflare.com/api/operations/queue-v...

Jason Barry•9/19/24, 2:42 PM

ah, I forgot to mention that I was trying to create queue consumers for user-uploaded scripts. I filed https://github.com/cloudflare/workers-sdk/issues/6758

JJason Barry ah, I forgot to mention that I was trying to create queue consumers for user-upl...

Hello, I’m Allie!•9/19/24, 2:44 PM

Via WfP?

HHello, I’m Allie!Via WfP?

Jason Barry•9/19/24, 2:44 PM

yep

JJason Barry yep

Hard@Work•9/19/24, 3:22 PM

Can you not just have the dispatcher call the user Worker?

Hard@Work•9/19/24, 3:23 PM

Also, I don't think WfP actually supports any kind of events other than one's sent by the dispatcher atm

HHard@Work Can you not just have the dispatcher call the user Worker?

Jason Barry•9/19/24, 3:32 PM

that’s the direction I was heading, by passing messages received in the dispatcher’s queue consumer to the user worker via a POST fetch. but how can the user worker call message.retry(), since the messages will be serialized into JSON?

JJason Barry that’s the direction I was heading, by passing messages received in the dispatch...

Hard@Work•9/19/24, 3:35 PM

It returns an object back with which messages to retry/ack