Just got `Queue send failed: Queue is overloaded. Please back off.` for the first time. Does that o

L

Lloyd•3/22/24, 11:07 AM

How comsumer works if income message over 100per seconds?
case 1 or case 2?

L

Lloyd•3/22/24, 11:10 AM

oh i see. thankyou

JJohn Spurlock Just got `Queue send failed: Queue is overloaded. Please back off.` for the firs...

M

Matt Silverlock•3/22/24, 2:01 PM

You should get

Too Many Requests

Too Many Requests

Too Many Requests

Too Many Requests when you hit the rate limit. Did you exceed your backlog in a single queue?

MMatt Silverlock You should get `Too Many Requests` when you hit the rate limit. Did you exceed y...

M

Matt Silverlock•3/22/24, 2:02 PM

MMatt Silverlock Click to see attachment

J

John SpurlockOP•3/22/24, 3:36 PM

curious - the backlog looked normal, and rates never get close to the limit (see weekly charts) - this is a constant process that's been running for over a week now with little variation.

Queue send failed: Queue is overloaded. Please back off.

Queue send failed: Queue is overloaded. Please back off.

was received from workers all over the world at around the same time (calling

.send

.send

on a queue).

Almost feels like an error message that would come from a DO request queue overload. Would it be possible for your team to track down what causes this message? I don't see any clues in workerd. Perhaps there can be an internal backup in whatever internal queue magic sits behind

.send

.send

and the queue storage?

Ideally from an abstraction pov,

.send

.send

would only fail when exceeding the msg/s limits (globally) or some truly edge case node failure where it couldn't even save to any sort of recoverable internal storage.

JJohn Spurlock curious - the backlog looked normal, and rates never get close to the limit (see...

M

Matt Silverlock•3/22/24, 3:38 PM

(queueId, accountId) - ? We'll take a look...

MMatt Silverlock (queueId, accountId) - ? We'll take a look...

M

Matt Silverlock•3/22/24, 3:39 PM

(I should just add yours to my keyboard text expander shortcuts...)

FFedera How are people looping the incoming batches from the queue?

J

John SpurlockOP•3/24/24, 3:51 PM

for (const msg of batch.messages) {

for (const msg of batch.messages) {

for (const msg of batch.messages) {

for (const msg of batch.messages) {

FFedera How are people looping the incoming batches from the queue?

M

Matt Silverlock•3/25/24, 1:04 PM

https://developers.cloudflare.com/queues/reference/javascript-apis/#consumer

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

M

Matt Silverlock•3/25/24, 5:48 PM

@Queues - You can now configure pull-based consumers (aka "HTTP Pull") in Queues! Pull-consumers can run outside of Cloudflare, pull + ack over HTTP, and scale out concurrently.

Example:

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

➜  curl "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/queues/${QUEUE_ID}/messages/pull" -X POST --data '{}' \
     -H "Authorization: Bearer ${QUEUES_TOKEN}" \
     -H "Content-Type:application/json"

Docs: https://developers.cloudflare.com/queues/reference/pull-consumers/
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#support-for-pull-based-consumers

S

switz•3/25/24, 9:42 PM

If anyone forks the SQS consumer package for Cloudflare queues let me know

S

switz•3/25/24, 9:43 PM

https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer

J

johncantrell97•3/26/24, 12:33 AM

any chance queues makes it way into the free plan at all? similar to D1?

C

cg•3/26/24, 3:21 AM

hey all

C

cg•3/26/24, 3:21 AM

am I able to get an assist from a CF team member to clear a Queue

C

cg•3/26/24, 3:21 AM

I have a backlog of 32.48M messages

C

cg•3/26/24, 5:35 AM

ah, I just deleted the queue

Z

zegevlier•3/26/24, 5:21 PM

I can't comment on that limit, but you could always temporarily store the message somewhere, then put an identifier in the queue and retrieve that identifier on the queue event.

Original message was deleted

M

Matt Silverlock•3/26/24, 6:03 PM

Fixed for now.

M

Matt Silverlock•3/26/24, 6:04 PM

@Queues We've just shipped the ability to delay messages - both when writing to a queue as well as when retrying. This can also be set at the queue and/or consumer level as a default delay that applies to all messages.

Sending:

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

// Delay a singular message by 600 seconds (10 minutes)
await env.YOUR_QUEUE.send(message, { delaySeconds: 600 })

Retrying:

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

export default {
  async queue(batch: MessageBatch, env: Env, ctx: ExecutionContext) {
    for (const msg of batch.messages) {
      // Mark for retry and delay a singular message
      // by 3600 seconds (1 hour)
      msg.retry({delaySeconds: 3600})

    }
  },
};

Docs: https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages
Changelog: https://developers.cloudflare.com/queues/platform/changelog/#delay-messages-published-to-a-queue

Original message was deleted

M

Matt Silverlock•3/26/24, 6:04 PM

You can also split the message and write a batch, or write to R2 and use the upcoming Event Notifications feature to trigger an "event" when you write to R2 into a Queue so you can post-process the data.

Rrelero90 I am running into a challenge with TypeScript types and setting up a Queue produ...

M

Matt Silverlock•3/26/24, 6:14 PM

add()

add()

add()

add() is not a method on a Queue. Where are you seeing this?

add()

add()

add()

add() is not mentioned in the tutorial.

send()

send()

send()

send() is what you want to call.

Tutorial: https://developers.cloudflare.com/queues/get-started/
API docs: https://developers.cloudflare.com/queues/reference/javascript-apis/#producer

Cloudflare Docs

Get started · Cloudflare Queues

Cloudflare Queues is a flexible messaging queue that allows you to queue messages for asynchronous processing. By following this guide, you will …

Cloudflare Docs

Cloudflare Queues - JavaScript APIs · Cloudflare Queues

Cloudflare Queues is integrated with Cloudflare Workers. To send and receive messages, you must use a Worker.

Original message was deleted

M

Matt Silverlock•3/26/24, 7:20 PM

Good things take time

We'll have more updates to share publicly next week.

Original message was deleted

M

Matt Silverlock•3/26/24, 8:47 PM

Not at the moment, no.

U

Unsmart•3/26/24, 9:16 PM

I'd hope your payment processor would have deduplication on their end...

U

Unsmart•3/26/24, 9:51 PM

[[env.staging.queues.producers]]

Original message was deleted

W

Walshy•3/26/24, 9:54 PM

Yeah you'll need [[env.staging.queue.producers]]

W

Walshy•3/26/24, 9:54 PM

TOML is not a great config language imo :meowupsidedown:

FFlorian That's a starting point! There have been multiple discussions around delays and ...

M

Matt Silverlock•3/27/24, 1:01 AM

You can apply an exponential back off on each attempt - we’ll soon be exposing the # of attempts as part of the message metadata. With that and the ability to set a delay you can write your own back off alg.

V

Veld•3/27/24, 7:06 PM

oh dang

V

Veld•3/27/24, 7:06 PM

cloudflare to aws any% speedrun

K

kamama•3/28/24, 6:53 AM

Hi folks, is queue not supporting multiple workers in local?
lets say
worker1 : producer
worker2: consumer?

Kkamama Hi folks, is queue not supporting multiple workers in local? lets say worker1 : ...

M

Matt Silverlock•3/28/24, 4:09 PM

https://developers.cloudflare.com/queues/reference/local-development/#known-issues

Cloudflare Docs

Local Development · Cloudflare Queues

Queues support local development workflows using Wrangler, the command-line interface for Workers. Wrangler runs the same version of Queues as …

Original message was deleted

M

Matt Silverlock•3/29/24, 2:44 PM

https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messages

Cloudflare Docs

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

MMatt Silverlock https://developers.cloudflare.com/queues/reference/batching-retries/#delay-messa...

M

Matt Silverlock•3/29/24, 2:45 PM

S

switz•3/29/24, 9:27 PM

@Matt interestingly, looks like SQS will wait a short amount of time before responding if there are no messages (

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

 Pull consumers are designed to use a "short polling" approach, this means that the API from CloudFlare will respond immediately with any messages that are available, or an empty response if there are no messages available, this is different from SQS will wait an amount of time before responding with an empty response.

). is that something that http pull would benefit from here? or are you of the opinion that fast-response > quantity of requests is more important.

S

switz•3/29/24, 9:28 PM

nicholas is making a lot of progress over at https://github.com/bbc/cloudflare-queue-consumer

can't wait to start using it as it stabilizes

GitHub

GitHub - bbc/cloudflare-queue-consumer: Build CloudFlare Queue base...

Build CloudFlare Queue based applications without the boilerplate (based on SQS Consumer) - bbc/cloudflare-queue-consumer

Sswitz @Matt interestingly, looks like SQS will wait a short amount of time before resp...

M

Matt Silverlock•3/29/24, 10:06 PM

SQS supports two modes: long and short. Right now we support what’s typically known as short - which is more aligned with the idea of “pulling” when you’re ready. TBD on whether long pulling makes sense vs other work.

S

switz•3/29/24, 10:30 PM

another request, perhaps should be over at #r2, but would be nice to send r2 lifecycle events to a cloudflare queue

S

switz•3/29/24, 10:31 PM

https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Amazon S3 Event Notifications - Amazon Simple Storage Service

Set up and configure notifications so that key events on buckets cause a message to be sent to an Amazon SNS topic.

M

mau•3/30/24, 11:11 AM

Trying to implement queues in nextjs, not able to find any example. Does anyone has any poc?

Mmau Trying to implement queues in nextjs, not able to find any example. Does anyone ...

M

Matt Silverlock•3/30/24, 2:15 PM

https://developers.cloudflare.com/pages/framework-guides/nextjs/deploy-a-nextjs-site/#use-bindings-in-your-nextjs-application - replace D1/KV with Queues here.

Cloudflare Docs

Full-stack deployment · Cloudflare Pages docs

Deploy a full-stack Next.js site (recommended).

M

mau•3/30/24, 5:55 PM

Thanks @Matt for reply. I am not able to find the code implementation for nextjs. I did figure out the bindings part. For example:
async queue(){} there is nothing like that in nextjs… so how to hook that in nextjs. Thanks

Mmau Thanks @Matt for reply. I am not able to find the code implementation for nextjs...

M

Matt Silverlock•3/30/24, 6:02 PM

Your queue consumer won’t be part of your Next.js app - that’s not how your app would work. You write to the queue from your Next.js app. This isn’t a limitation of Queues - your Next.js app doesn’t have a concept of a “queue consumer” here since it’s request-response driven.

What are you trying to use a queue for here?

Just got `Queue send failed: Queue is overloaded. Please back off.` for the first time. Does that o

Similar Threads

Similar Threads

Similar Threads