Cloudflare Developers•2y ago

https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages

Cloudflare Docs

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messa...

Matt SilverlockOP•3/29/24, 2:45 PM

switz•3/29/24, 9:27 PM

@Matt interestingly, looks like SQS will wait a short amount of time before responding if there are no messages (

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

). is that something that http pull would benefit from here? or are you of the opinion that fast-response > quantity of requests is more important.

switz•3/29/24, 9:28 PM

nicholas is making a lot of progress over at https://github.com/bbc/cloudflare-queue-consumer

can't wait to start using it as it stabilizes

GitHub

GitHub - bbc/cloudflare-queue-consumer: Build CloudFlare Queue base...

Build CloudFlare Queue based applications without the boilerplate (based on SQS Consumer) - bbc/cloudflare-queue-consumer

Sswitz @Matt interestingly, looks like SQS will wait a short amount of time before resp...

Matt SilverlockOP•3/29/24, 10:06 PM

SQS supports two modes: long and short. Right now we support what’s typically known as short - which is more aligned with the idea of “pulling” when you’re ready. TBD on whether long pulling makes sense vs other work.

switz•3/29/24, 10:30 PM

another request, perhaps should be over at #r2, but would be nice to send r2 lifecycle events to a cloudflare queue

switz•3/29/24, 10:31 PM

https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Amazon S3 Event Notifications - Amazon Simple Storage Service

Set up and configure notifications so that key events on buckets cause a message to be sent to an Amazon SNS topic.

mau•3/30/24, 11:11 AM

Trying to implement queues in nextjs, not able to find any example. Does anyone has any poc?

Mmau Trying to implement queues in nextjs, not able to find any example. Does anyone ...

Matt SilverlockOP•3/30/24, 2:15 PM

https://developers.cloudflare.com/pages/framework-guides/nextjs/deploy-a-nextjs-site/#use-bindings-in-your-nextjs-application

replace D1/KV with Queues here.

Cloudflare Docs

Full-stack deployment · Cloudflare Pages docs

Deploy a full-stack Next.js site (recommended).

mau•3/30/24, 5:55 PM

Thanks @Matt for reply. I am not able to find the code implementation for nextjs. I did figure out the bindings part. For example:
async queue(){} there is nothing like that in nextjs… so how to hook that in nextjs. Thanks

Mmau Thanks @Matt for reply. I am not able to find the code implementation for nextjs...

Matt SilverlockOP•3/30/24, 6:02 PM

Your queue consumer won’t be part of your Next.js app - that’s not how your app would work. You write to the queue from your Next.js app. This isn’t a limitation of Queues - your Next.js app doesn’t have a concept of a “queue consumer” here since it’s request-response driven.

What are you trying to use a queue for here?

mau•3/31/24, 5:07 AM

I am developing an e-learning application and focusing on optimizing data processing. Our system evaluates quiz answers in real-time, providing immediate feedback. We store the quiz progress and answers in MongoDB, allowing students to resume their work without any issues.However, we face a challenge with the delay users experience at the result screen. This delay occurs because, currently, when a student submits a quiz, we transfer the data from MongoDB to PostgreSQL. This data transfer and processing happen in the same route as the quiz submission, leading to a wait time for the user to see their results. To mitigate this, we are considering implementing queues to handle the data transfer process separately, thereby reducing the load during submission and speeding up the display of results. Additionally, we are exploring the use of key-value stores or durable objects to further optimize this workflow.

mau•3/31/24, 5:08 AM

We are trying to use all the products provided by Cloudflare so that we can build highly Scaleable product.

Mmau I am developing an e-learning application and focusing on optimizing data proces...

Matt SilverlockOP•3/31/24, 3:33 PM

It’s hard to fully understand your architecture at this level, but if you want to make your application more real time, I’m not sure how Queues would do that. Using a Durable Object or just D1 would likely be better - no need to “sync” or wait on data? It’s a quiz. It should be immediate.

mau•3/31/24, 3:36 PM

Result will be still immediate, it’s just I was thinking to build a asynchronous way of moving data from mongo (which is used to store data while users are working on the quiz) to Postgres’s (which is final destination)… I’m also exploring d1 (because I need something which allows unstructured data and it’s quick which will replace mongo).

Unsmart•3/31/24, 5:54 PM

I’m confused why use multiple databases like that for the same data

Mmau Result will be still immediate, it’s just I was thinking to build a asynchronous...

Matt SilverlockOP•3/31/24, 10:02 PM

Right, if the user response does NOT need to block on that data movement, then Queues can be used as a way to batch up changes and then write them to Postgres from your consumer.

You would bind to the Queue and call send()send() within your Next.js app exactly like the Pages tutorial I sent you.

Your consumer

the part that consumes the messages and writes to Postgres - would just be a standalone Worker.

Matt SilverlockOP•4/1/24, 1:34 PM

@Queues - in case you missed it, Queues now supports pull-based consumers (pull from any HTTP client!) and fine-grained controls over delaying messages: https://blog.cloudflare.com/making-full-stack-easier-d1-ga-hyperdrive-queues (live as of last week)

The Cloudflare Blog

Making state easy with D1 GA, Hyperdrive, Queues and Workers Analyt...

We kick off the week with announcements that help developers build stateful applications on top of Cloudflare, including making D1, our SQL database and Hyperdrive, our database accelerating service, generally available

kp1197•4/1/24, 8:47 PM

Any general impressions on stsbility of this feature?

boywonder350•4/1/24, 10:58 PM

How long can you delay a message for?

boywonder350•4/1/24, 10:58 PM

Say I want to delay a message for 1 year, is that possible? (1 year subscription renewal reminder for example)

Bboywonder350 Say I want to delay a message for 1 year, is that possible? (1 year subscription...

Chaika•4/1/24, 11:50 PM

The max delaySeconds is 12 hours: https://developers.cloudflare.com/queues/platform/limits/

Cloudflare Docs

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

Bboywonder350 Say I want to delay a message for 1 year, is that possible? (1 year subscription...

Isaac McFadyen•4/2/24, 3:45 AM

In general you don't want to have a delay for this very long-term type of thing (i.e. what if the user deletes their account? You don't want to be scheduling a task for one year and then sending a deleted user a reminder email).

Isaac McFadyen•4/2/24, 3:45 AM

Instead the general way this is done is by setting a task that runs every hour (for example, with Workers Cron events or another service) that checks a database for any tasks that need to be completed, runs them, and then sets them to Completed.

Isaac McFadyen•4/2/24, 3:45 AM

That way, if the user deletes their account or you need to cancel the task for any reason you just delete any tasks associated with that user.

Sswitz another request, perhaps should be over at #r2, but would be nice to send r2 lif...

switz•4/3/24, 4:06 PM

Let’s gooo!!!! https://x.com/ritakozlov_/status/1775514140229968041?s=46&t=_hz7_TqpYWiUUE4FPGb-5Q

rita kozlov 🐀 (@ritakozlov_) on X

i’m excited for today’s announcements and i hope you r2!

r2 event notifications are here! this is a big deal — the beginning of events for workers

gcs data migrator is here (if you want real zero-egress storage)

infrequent access tiers for r2 beta…

Twitter

•

4/3/24, 1:22 PM

Sswitz Let’s gooo!!!! https://x.com/ritakozlov_/status/1775514140229968041?s=46&t=_hz7_...

Matt SilverlockOP•4/3/24, 4:11 PM

$ npx wrangler@latest r2 bucket event-notification create user-media-prod --event-type="object_create" --queue="prod-upload-events"

$ npx wrangler@latest r2 bucket event-notification create user-media-prod --event-type="object_create" --queue="prod-upload-events"

switz•4/3/24, 4:11 PM

This is fantastic, thank you. The final missing piece for me to leave s3 and SQS entirely

switz•4/3/24, 4:11 PM

Is there also object expire?

switz•4/3/24, 4:14 PM

There is object delete :cool_doge:

switz•4/3/24, 4:14 PM

“Note: During the beta, deletes that occur as a result of object lifecycle policies will not trigger this event”

Ahhhh crap. Why is there always a caveat lol

switz•4/3/24, 4:15 PM

That’s a blocker

squareclamp•4/3/24, 4:20 PM

Any tips on how I can investigate why some messages in queues are listed in dashboard but aren't being processed? I suppose maybe they have delay - I can't confirm dashboard but I also don't remember configuring delay to several hours.

squareclamp•4/3/24, 4:25 PM

Looking at the chrome isnpector, the messages have delay < 1 hour - but its been several hours messages aren't processed.

"metadata": {
                    "CF-Content-Type": "json",
                    "CF-msg-delay-secs": "3200"
                }

"metadata": {
                    "CF-Content-Type": "json",
                    "CF-msg-delay-secs": "3200"
                }

squareclamp•4/3/24, 4:29 PM

My suspect is maybe having retry in code with max retries set to zero in config.. but changing max retry config hasn't helped.

jgoux•4/3/24, 8:31 PM

Hello team, is there an equivalent to SNS in Cloudflare? I'd like to implement a fanout architecture where I can push a payload and have multiple couples [Queue + Worker] consuming it.

Jjgoux Hello team, is there an equivalent to SNS in Cloudflare? I'd like to implement a...

avsaase•4/3/24, 9:00 PM

Sounds like you need pubsub which is in private beta.

Aavsaase Sounds like you need pubsub which is in private beta.

jgoux•4/3/24, 9:02 PM

The Developer Week isn't over, I have hope for a public beta then.

LordSilver•4/3/24, 9:06 PM

Isn't Queues GA? Why the channel still has

beta

beta

in the name?

LLordSilver Isn't Queues GA? Why the channel still has `beta` in the name?

John Spurlock•4/3/24, 9:32 PM

queues is not GA yet

https://blog.cloudflare.com/making-full-stack-easier-d1-ga-hyperdrive-queues/

JJohn Spurlock queues is not GA yet https://blog.cloudflare.com/making-full-stack-easier-d1-ga...

LordSilver•4/3/24, 9:49 PM

It doesn't have a DLQ?

LLordSilver It doesn't have a DLQ?

John Spurlock•4/3/24, 9:53 PM

Queues has Dead Letter Queues, did you read the docs?

https://developers.cloudflare.com/queues/reference/dead-letter-queues/

Cloudflare Docs

Dead Letter Queues · Cloudflare Queues

A Dead Letter Queue (DLQ) is a common concept in a messaging system, and represents where messages are sent when a delivery failure occurs with a …

LordSilver•4/3/24, 10:36 PM

Why you need to have a sub even if you just want to test the free tier?

Unsmart•4/4/24, 12:07 AM

The included tier is only available in the workers paid tier. (You need to pay for workers to get access to queues at all)

Patrick M•4/4/24, 8:47 AM

Yeah I got confused too, I feel like the phrase “Free Tier” is kind of a misnomer https://developers.cloudflare.com/queues/platform/pricing/

Cloudflare Docs

Cloudflare Queues - Pricing · Cloudflare Queues

Cloudflare Queues requires the Workers Paid plan to use, but does not increase your monthly subscription cost.

PPatrick M Yeah I got confused too, I feel like the phrase “Free Tier” is kind of a misnome...

Sam•4/4/24, 8:48 AM

It literally says "Cloudflare Queues requires the Workers Paid plan to use, but does not increase your monthly subscription cost." at the top of this page lmao

Patrick M•4/4/24, 8:51 AM

Yes I know, I read that and though it was “paid” from reading articles before the I saw this page too. Then I saw “free tier” and thought my understanding was wrong. Turns out I understood it correctly. I think it would be easier to understand if they called it “included in workers paid” or something instead of “free tier”.

switz•4/4/24, 4:15 PM

I use queues on a free plan? :Shrug:

switz•4/4/24, 4:15 PM

maybe I opted in early enough?

Unsmart•4/4/24, 4:46 PM

You have a workers paid subscription

switz•4/4/24, 6:50 PM

is there a way to send data into a queue via the REST api or do I have to write a worker to wrap it?

https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages

Similar Threads