curious - the backlog looked normal, and rates never get close to the limit (see weekly charts) - th

curious - the backlog looked normal, and rates never get close to the limit (see weekly charts) - this is a constant process that's been running for over a week now with little variation.

Queue send failed: Queue is overloaded. Please back off.

Queue send failed: Queue is overloaded. Please back off.

was received from workers all over the world at around the same time (calling

.send

.send

on a queue).

Almost feels like an error message that would come from a DO request queue overload. Would it be possible for your team to track down what causes this message? I don't see any clues in workerd. Perhaps there can be an internal backup in whatever internal queue magic sits behind

.send

.send

and the queue storage?

Ideally from an abstraction pov,

.send

.send

would only fail when exceeding the msg/s limits (globally) or some truly edge case node failure where it couldn't even save to any sort of recoverable internal storage.

JJohn Spurlock curious - the backlog looked normal, and rates never get close to the limit (see...

Matt Silverlock•3/22/24, 3:38 PM

(queueId, accountId) - ? We'll take a look...

MMatt Silverlock (queueId, accountId) - ? We'll take a look...

Matt Silverlock•3/22/24, 3:39 PM

(I should just add yours to my keyboard text expander shortcuts...)

FFedera How are people looping the incoming batches from the queue?

John SpurlockOP•3/24/24, 3:51 PM

for (const msg of batch.messages) {

for (const msg of batch.messages) {

FFedera How are people looping the incoming batches from the queue?

Matt Silverlock•3/25/24, 1:04 PM

https://developers.cloudflare.com/queues/reference/javascript-apis/#consumer

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Matt Silverlock•3/25/24, 5:48 PM

@Queues - You can now configure pull-based consumers (aka "HTTP Pull") in Queues! Pull-consumers can run outside of Cloudflare, pull + ack over HTTP, and scale out concurrently.

Example:

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

Docs: https://developers.cloudflare.com/queues/reference/pull-consumers/
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#support-for-pull-based-consumers

switz•3/25/24, 9:42 PM

If anyone forks the SQS consumer package for Cloudflare queues let me know

switz•3/25/24, 9:43 PM

https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer

johncantrell97•3/26/24, 12:33 AM

any chance queues makes it way into the free plan at all? similar to D1?

cg•3/26/24, 3:21 AM

hey all

cg•3/26/24, 3:21 AM

am I able to get an assist from a CF team member to clear a Queue

cg•3/26/24, 3:21 AM

I have a backlog of 32.48M messages

cg•3/26/24, 5:35 AM

ah, I just deleted the queue

zegevlier•3/26/24, 5:21 PM

I can't comment on that limit, but you could always temporarily store the message somewhere, then put an identifier in the queue and retrieve that identifier on the queue event.

Original message was deleted

Matt Silverlock•3/26/24, 6:03 PM

Fixed for now.

Matt Silverlock•3/26/24, 6:04 PM

@Queues We've just shipped the ability to delay messages - both when writing to a queue as well as when retrying. This can also be set at the queue and/or consumer level as a default delay that applies to all messages.

Sending:

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

Retrying:

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

Docs: https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#delay-messages-published-to-a-queue

Original message was deleted

Matt Silverlock•3/26/24, 6:04 PM

You can also split the message and write a batch, or write to R2 and use the upcoming Event Notifications feature to trigger an "event" when you write to R2 into a Queue so you can post-process the data.

Rrelero90 I am running into a challenge with TypeScript types and setting up a Queue produ...

Matt Silverlock•3/26/24, 6:14 PM

add()

add()

is not a method on a Queue. Where are you seeing this?

add()

add()

is not mentioned in the tutorial.

send()

send()

is what you want to call.

Tutorial: https://developers.cloudflare.com/queues/get-started/
API docs: https://developers.cloudflare.com/queues/reference/javascript-apis/#producer

Cloudflare Docs

Get started · Cloudflare Queues

Cloudflare Queues is a flexible messaging queue that allows you to queue messages for asynchronous processing. By following this guide, you will …

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Original message was deleted

Matt Silverlock•3/26/24, 7:20 PM

Good things take time

We'll have more updates to share publicly next week.

Original message was deleted

Matt Silverlock•3/26/24, 8:47 PM

Not at the moment, no.

Unsmart•3/26/24, 9:16 PM

I'd hope your payment processor would have deduplication on their end...

Unsmart•3/26/24, 9:51 PM

[[env.staging.queues.producers]]

Original message was deleted

Walshy•3/26/24, 9:54 PM

Yeah you'll need [[env.staging.queue.producers]]

Walshy•3/26/24, 9:54 PM

TOML is not a great config language imo :meowupsidedown:

FFlorian That's a starting point! There have been multiple discussions around delays and ...

Matt Silverlock•3/27/24, 1:01 AM

You can apply an exponential back off on each attempt - we’ll soon be exposing the # of attempts as part of the message metadata. With that and the ability to set a delay you can write your own back off alg.

Veld•3/27/24, 7:06 PM

oh dang

Veld•3/27/24, 7:06 PM

cloudflare to aws any% speedrun

kamama•3/28/24, 6:53 AM

Hi folks, is queue not supporting multiple workers in local?
lets say
worker1 : producer
worker2: consumer?

Kkamama Hi folks, is queue not supporting multiple workers in local? lets say worker1 : ...

Matt Silverlock•3/28/24, 4:09 PM

https://developers.cloudflare.com/queues/reference/local-development/#known-issues

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

Original message was deleted

Matt Silverlock•3/29/24, 2:44 PM

https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages

Cloudflare Docs

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messa...

Matt Silverlock•3/29/24, 2:45 PM

switz•3/29/24, 9:27 PM

@Matt interestingly, looks like SQS will wait a short amount of time before responding if there are no messages (

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

). is that something that http pull would benefit from here? or are you of the opinion that fast-response > quantity of requests is more important.

switz•3/29/24, 9:28 PM

nicholas is making a lot of progress over at https://github.com/bbc/cloudflare-queue-consumer

can't wait to start using it as it stabilizes

GitHub

GitHub - bbc/cloudflare-queue-consumer: Build CloudFlare Queue base...

Build CloudFlare Queue based applications without the boilerplate (based on SQS Consumer) - bbc/cloudflare-queue-consumer

Sswitz @Matt interestingly, looks like SQS will wait a short amount of time before resp...

Matt Silverlock•3/29/24, 10:06 PM

SQS supports two modes: long and short. Right now we support what’s typically known as short - which is more aligned with the idea of “pulling” when you’re ready. TBD on whether long pulling makes sense vs other work.

switz•3/29/24, 10:30 PM

another request, perhaps should be over at #r2, but would be nice to send r2 lifecycle events to a cloudflare queue

switz•3/29/24, 10:31 PM

https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Amazon S3 Event Notifications - Amazon Simple Storage Service

Set up and configure notifications so that key events on buckets cause a message to be sent to an Amazon SNS topic.

mau•3/30/24, 11:11 AM

Trying to implement queues in nextjs, not able to find any example. Does anyone has any poc?

Mmau Trying to implement queues in nextjs, not able to find any example. Does anyone ...

Matt Silverlock•3/30/24, 2:15 PM

https://developers.cloudflare.com/pages/framework-guides/nextjs/deploy-a-nextjs-site/#use-bindings-in-your-nextjs-application - replace D1/KV with Queues here.

Cloudflare Docs

Full-stack deployment · Cloudflare Pages docs

Deploy a full-stack Next.js site (recommended).

mau•3/30/24, 5:55 PM

Thanks @Matt for reply. I am not able to find the code implementation for nextjs. I did figure out the bindings part. For example:
async queue(){} there is nothing like that in nextjs… so how to hook that in nextjs. Thanks

Mmau Thanks @Matt for reply. I am not able to find the code implementation for nextjs...

Matt Silverlock•3/30/24, 6:02 PM

Your queue consumer won’t be part of your Next.js app - that’s not how your app would work. You write to the queue from your Next.js app. This isn’t a limitation of Queues - your Next.js app doesn’t have a concept of a “queue consumer” here since it’s request-response driven.

What are you trying to use a queue for here?

mau•3/31/24, 5:07 AM

I am developing an e-learning application and focusing on optimizing data processing. Our system evaluates quiz answers in real-time, providing immediate feedback. We store the quiz progress and answers in MongoDB, allowing students to resume their work without any issues.However, we face a challenge with the delay users experience at the result screen. This delay occurs because, currently, when a student submits a quiz, we transfer the data from MongoDB to PostgreSQL. This data transfer and processing happen in the same route as the quiz submission, leading to a wait time for the user to see their results. To mitigate this, we are considering implementing queues to handle the data transfer process separately, thereby reducing the load during submission and speeding up the display of results. Additionally, we are exploring the use of key-value stores or durable objects to further optimize this workflow.

mau•3/31/24, 5:08 AM

We are trying to use all the products provided by Cloudflare so that we can build highly Scaleable product.

Mmau I am developing an e-learning application and focusing on optimizing data proces...

Matt Silverlock•3/31/24, 3:33 PM

It’s hard to fully understand your architecture at this level, but if you want to make your application more real time, I’m not sure how Queues would do that. Using a Durable Object or just D1 would likely be better - no need to “sync” or wait on data? It’s a quiz. It should be immediate.

mau•3/31/24, 3:36 PM

Result will be still immediate, it’s just I was thinking to build a asynchronous way of moving data from mongo (which is used to store data while users are working on the quiz) to Postgres’s (which is final destination)… I’m also exploring d1 (because I need something which allows unstructured data and it’s quick which will replace mongo).

Unsmart•3/31/24, 5:54 PM

I’m confused why use multiple databases like that for the same data

curious - the backlog looked normal, and rates never get close to the limit (see weekly charts) - th

Similar Threads

Similar Threads

Similar Threads