Hello team, is there an equivalent to SNS in Cloudflare? I'd like to implement a fanout architecture

Hello team, is there an equivalent to SNS in Cloudflare? I'd like to implement a fanout architecture where I can push a payload and have multiple couples [Queue + Worker] consuming it.

Jjgoux Hello team, is there an equivalent to SNS in Cloudflare? I'd like to implement a...

avsaase•4/3/24, 9:00 PM

Sounds like you need pubsub which is in private beta.

Aavsaase Sounds like you need pubsub which is in private beta.

jgouxOP•4/3/24, 9:02 PM

The Developer Week isn't over, I have hope for a public beta then.

LordSilver•4/3/24, 9:06 PM

Isn't Queues GA? Why the channel still has

beta

beta

in the name?

LLordSilver Isn't Queues GA? Why the channel still has `beta` in the name?

John Spurlock•4/3/24, 9:32 PM

queues is not GA yet

https://blog.cloudflare.com/making-full-stack-easier-d1-ga-hyperdrive-queues/

JJohn Spurlock queues is not GA yet https://blog.cloudflare.com/making-full-stack-easier-d1-ga...

LordSilver•4/3/24, 9:49 PM

It doesn't have a DLQ?

LLordSilver It doesn't have a DLQ?

John Spurlock•4/3/24, 9:53 PM

Queues has Dead Letter Queues, did you read the docs?

https://developers.cloudflare.com/queues/reference/dead-letter-queues/

Cloudflare Docs

Dead Letter Queues · Cloudflare Queues

A Dead Letter Queue (DLQ) is a common concept in a messaging system, and represents where messages are sent when a delivery failure occurs with a …

LordSilver•4/3/24, 10:36 PM

Why you need to have a sub even if you just want to test the free tier?

Unsmart•4/4/24, 12:07 AM

The included tier is only available in the workers paid tier. (You need to pay for workers to get access to queues at all)

Patrick M•4/4/24, 8:47 AM

Yeah I got confused too, I feel like the phrase “Free Tier” is kind of a misnomer https://developers.cloudflare.com/queues/platform/pricing/

Cloudflare Docs

Cloudflare Queues - Pricing · Cloudflare Queues

Cloudflare Queues requires the Workers Paid plan to use, but does not increase your monthly subscription cost.

PPatrick M Yeah I got confused too, I feel like the phrase “Free Tier” is kind of a misnome...

Sam•4/4/24, 8:48 AM

It literally says "Cloudflare Queues requires the Workers Paid plan to use, but does not increase your monthly subscription cost." at the top of this page lmao

Patrick M•4/4/24, 8:51 AM

Yes I know, I read that and though it was “paid” from reading articles before the I saw this page too. Then I saw “free tier” and thought my understanding was wrong. Turns out I understood it correctly. I think it would be easier to understand if they called it “included in workers paid” or something instead of “free tier”.

switz•4/4/24, 4:15 PM

I use queues on a free plan? :Shrug:

switz•4/4/24, 4:15 PM

maybe I opted in early enough?

Unsmart•4/4/24, 4:46 PM

You have a workers paid subscription

switz•4/4/24, 6:50 PM

is there a way to send data into a queue via the REST api or do I have to write a worker to wrap it?

Sswitz is there a way to send data into a queue via the REST api or do I have to write ...

Matt Silverlock•4/4/24, 7:27 PM

You'll need to write a Worker: https://developers.cloudflare.com/queues/examples/publish-to-a-queue-over-http/

you can consume from any HTTP client, but you'll need a Worker to write to a Queue for now.

switz•4/4/24, 7:27 PM

yeah, that's what I did. would be nice if you could off the shelf with the REST api/cloudflare typescript package. but only took me ~10 mins to write one myself

switz•4/4/24, 11:00 PM

is there a way to add an idempotent id for a queue message to avoid duplicate messages for messages that may already be in a queue?

my use case is: occasionally if an external entity is not cleaned up, I may send a message to a queue to clean it up – but if that message hasn't been handled yet, I'd rather not continue filling up the queue with the same message.

alternatively, if there's a way to see if the queue has a message that would also be helpful.

kalepail•4/4/24, 11:09 PM

It seems that

delaySeconds

delaySeconds

doesn't work locally. Is this being tracked anywhere or should I open an issue?

maciej•4/5/24, 11:12 AM

Hey, what are the plans around supporting Pages Functions as queue consumers? Is it something on the roadmap?

Kkalepail It seems that `delaySeconds` doesn't work locally. Is this being tracked anywher...

se•4/5/24, 3:09 PM

We are aware of that and it's being tracked. We have WIP for that.

Bryan•4/5/24, 5:00 PM

Is there any plan for supporting message de-duplicate?

Sometimes I want to retry sending a message, but I don't want it to be handled twice more. Many message queue can set a custom MessageId for message de-duplication. Will queues support it?

Unsmart•4/5/24, 6:00 PM

You should always make your queue handler idempotent. It will always be possible that a single message be replayed twice (or more).

Victor•4/6/24, 9:00 PM

I found an weird inconsistency in batch timeout. Miniflare validator of

wrangler.toml

wrangler.toml

says

Number must be less than or equal to 30

Number must be less than or equal to 30

, the limits doc page (https://developers.cloudflare.com/queues/platform/limits/#limits) says max

, but the browser rendering example shows

https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-rendering/#3-configure-your-worker and the web gui says max

Cloudflare Docs

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

Cloudflare Docs

Cloudflare Queues - Queues & Browser Rendering · Cloudflare Queues

This tutorial explains how to build and deploy a web crawler with Queues, Browser Rendering, and Puppeteer.

VVictor I found an weird inconsistency in batch timeout. Miniflare validator of `wrangle...

Victor•4/6/24, 11:23 PM

Opened https://github.com/cloudflare/cloudflare-docs/issues/13894

GitHub

Queues Batch timeout inconsistencies · Issue #13894 · cloudflare/cl...

Existing documentation URL(s) https://developers.cloudflare.com/queues/platform/limits/#limits https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-rendering/#3-configure-you...

boywonder350•4/7/24, 3:26 AM

I am trying out the new r2 events with a queue and I am noticing it is pretty slow. My process is on photo upload, listen for the event in a queue and have a worker generate 10 variations with image transformation and then store the variations in r2 and then save all the info in a db. I stress tested the system by uploading 200 images (with 4 concurrently uploaded at a time) and noticed that after the images uploaded, the queue worker is still working on processing the events like 10 minutes later. Not sure if this is just a beta thing, but I was hoping this would have better results. Anyone know whats up with this?

Bboywonder350 I am trying out the new r2 events with a queue and I am noticing it is pretty sl...

Matt Silverlock•4/7/24, 10:23 AM

What do you mean by slow? Slow for it to receive the events in the first place, or to process them in your consumer? What are your batch settings?

Can you share your code if it’s the latter? The consumer is only going to be as fast as the code you write / the work it needs to do.

MMatt Silverlock What do you mean by slow? Slow for it to receive the events in the first place, ...

boywonder350•4/7/24, 1:18 PM

Also here is the stats for the worker processing it. Wall time means the amount of time it took the worker to process the message right? What is the max amount of workers that can take a queue message and process it? It feels like it is slow to process the events to spin up a worker.

Bboywonder350 Also here is the stats for the worker processing it. Wall time means the amount ...

Matt Silverlock•4/7/24, 4:25 PM

https://developers.cloudflare.com/queues/reference/consumer-concurrency/

Cloudflare Docs

Consumer concurrency · Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that …

MMatt Silverlock https://developers.cloudflare.com/queues/reference/consumer-concurrency/

boywonder350•4/7/24, 4:29 PM

Are there any available metrics to see how many consumer workers were used for a duration? Following the documentation, I am looking for max concurrency and I only find this:
"Concurrent consumer invocations 20 push-based only"

What does it mean to be "push-based only" here?

Bboywonder350 Are there any available metrics to see how many consumer workers were used for a...

Matt Silverlock•4/7/24, 7:34 PM

https://developers.cloudflare.com/queues/reference/how-queues-works/#consumers

Cloudflare Docs

How Queues Works · Cloudflare Queues

Cloudflare Queues is a flexible messaging queue that allows you to queue messages for asynchronous processing. Message queues are great at decoupling …

boywonder350•4/7/24, 7:54 PM

I guess the slowness was due to there only being 20 concurrent consumers. Will this ever change?

boywonder350•4/7/24, 7:54 PM

The issue is for one image upload, I then create 10 variations. I then need to no-op the consumer worker for those 10 items because I don’t want to process them

Bboywonder350 The issue is for one image upload, I then create 10 variations. I then need to n...

Matt Silverlock•4/7/24, 9:52 PM

Why not set up Event Notifications to only send you events for objects matching a certain prefix, or write to a different bucket, to avoid handling events you don’t want to process? Otherwise just

ack()

ack()

those messages and move on.

boywonder350•4/7/24, 9:56 PM

ooo totally fixxed the prefix or suffix documentation!

DanGamble•4/8/24, 9:42 AM

Given you can't use a pages app and a separate worker to consume a queue locally what is people's current workflows for handling this? I was thinking of deploying a queue and then just using that but I'm not sure how to have 1 binding be remote but the rest local

serkanozal•4/8/24, 6:28 PM

Hi everyone,

I am experimenting with Cloudflare Workers and Queues. During my experiments, I have noticed that sending a simple message from my Cloudflare Worker to a binded Cloudflare Queue takes about 200 ms in the warm invocations. Is this expected? Might it be caused by network latency if the binded queue is used from another data center or is it just Cloudflare Queue slow?

Especially comparing with AWS SQS send message latency, it seems too much to me. Because according to my experiments, it is around 10 ms for sending a message to AWS SQS queue, but it is 200 ms for Cloudflare Queue.

Original message was deleted

Matt Silverlock•4/8/24, 7:40 PM

It should look like this: https://developers.cloudflare.com/api/operations/queue-v2-messages-ack

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Original message was deleted

Matt Silverlock•4/8/24, 8:07 PM

Can you confirm the path you’re hitting, just to be sure?

Original message was deleted

Matt Silverlock•4/8/24, 8:17 PM

The undocumented one is deprecated (and will be removed in future). Let me add a note to the docs just to be sure.

Original message was deleted

Matt Silverlock•4/8/24, 8:39 PM

https://github.com/cloudflare/cloudflare-docs/pull/13924

GitHub

queues: clarify undocumented endpoints by elithrar · Pull Request #...

cc @w-kuhn

John Spurlock•4/9/24, 11:07 PM

did queues conk out for about 5 minutes last hour? No messages delivered to

queue

queue

from around 5.23pm to 5.28pm us central

MMatt Silverlock You can apply an exponential back off on each attempt - we’ll soon be exposing t...

Matt Silverlock•4/10/24, 11:30 AM

@Florian the

attempts

attempts

field on messages has shipped. Docs PR (waiting on build): https://github.com/cloudflare/cloudflare-docs/pull/13648

GitHub

[QUEUES] Document how to apply a backoff alg by elithrar · Pull Req...

Documents how to apply a backoff alg when calling retry().

JJohn Spurlock did queues conk out for about 5 minutes last hour? No messages delivered to `qu...

Matt Silverlock•4/10/24, 1:05 PM

No widespread issues - queue ID?

Hhannes afaik queues run only in loctions where DOs run, which is like 12% of all Cloudf...

serkanozal•4/10/24, 5:36 PM

Thanks for the detailed answer. Is there any way to verify this (location of the queue colo)?

Original message was deleted

Matt Silverlock•4/11/24, 10:18 PM

You can change the # count in the UI - right above the picker. There isn’t pagination though due to the way Queues works right now.

Hhannes Hrm the queue worker should be able to fetch “/cdn-cgi/trace” and parse the colo...

serkanozal•4/12/24, 9:20 AM

When I request to the "<MY-DOMAIN>/cdn-cgi/trace", the colo in the response is "FRA". But as far as I understood, it is the colo of my worker, but CloudFlare queue might be served from another colo. Or does it mean that the queue itself is used from colo "FRA" too? If so, why the latency is so high (200ms)?

Hello team, is there an equivalent to SNS in Cloudflare? I'd like to implement a fanout architecture

Similar Threads

Similar Threads

Similar Threads