The 400msg/s limit of a queue is bound to the single producer, correct? If I bind for example 10 que

Kkev-ac The 400msg/s limit of a queue is bound to the single producer, correct? If I bin...

P

the limit is per queue. so you can manually shard / distribute messages across multiple queues yes, to get higher throughput. It's what a lot of folks do today. Although... higher queue limits are coming very soon!

PPranshu Maheshwari the limit is per queue. so you can manually shard / distribute messages across m...

K

kev-acOP•7/26/24, 4:12 PM

Thanks for the clarification! What I did not notice before is that I'm also really limited by a separate factor here: The max. 6 outbound connection limit. The use case (votings) will produce multiple thousand messages in a minute or so. But the consumer side cannot push messages (in this case to an external backend) fast enough as it is limited to a maximum of 6 concurrent connections. And even worse it won't scale up the consumer concurrency from 1.

Kkev-ac Thanks for the clarification! What I did not notice before is that I'm also real...

P

Pranshu Maheshwari•7/26/24, 6:50 PM

ah gotcha, yeah the limit of 6 outbound connections is a hard one unfortunately.

Your consumers should scale up though; at the end of processing each batch of messages, queues checks if the consumer concurrency should be increased (based on looking at the backlog). There are a few reasons why it might not autoscale (listed on this page in the docs: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/)

Lmk in case you're having trouble with autoscaling and I can help (just send me over your account ID and queue ID, both are safe to share here)

0

0xthierry•7/26/24, 7:30 PM

hey guys, I'm using cf queue, but I'm experiencing some delay to delivery the message, after publishing the message in the queue, the worker starts to process it after +30s. Do you have any idea about it?

J

João Castro•7/27/24, 5:33 AM

Same here, sometimes i'm experiencing minutes of delay for messages to be consumed

Z

ZZA•7/29/24, 9:44 AM

Hello guys,
i'm trying to add event notifications to one of my buckets using the command

npx wrangler r2 bucket notification create <BUCKET_NAME> --event-type <EVENT_TYPE> --queue <QUEUE_NAME>

npx wrangler r2 bucket notification create <BUCKET_NAME> --event-type <EVENT_TYPE> --queue <QUEUE_NAME>

npx wrangler r2 bucket notification create <BUCKET_NAME> --event-type <EVENT_TYPE> --queue <QUEUE_NAME>

npx wrangler r2 bucket notification create <BUCKET_NAME> --event-type <EVENT_TYPE> --queue <QUEUE_NAME>

i'm getting a bucket not found error, because my bucket's jurisdiction is EU. but i can't see to find how i can specify that to

wrangler r2 bucket notification

wrangler r2 bucket notification

wrangler r2 bucket notification

wrangler r2 bucket notification

any one has an idea how i can get this to possibly work ?

ZZZA Hello guys, i'm trying to add event notifications to one of my buckets using th...

K

kev-acOP•7/29/24, 10:33 AM

Don't quote me on that but it may be possible that event notifications don't work on jurisdictional buckets yet. Super Slurper and Logpush are not supported right now, so it may also apply to Event notifications.

00xthierry hey guys, I'm using cf queue, but I'm experiencing some delay to delivery the me...

P

Pranshu Maheshwari•7/29/24, 3:30 PM

Could you send me your queue ID & acct ID? both are safe to share here

PPranshu Maheshwari Could you send me your queue ID & acct ID? both are safe to share here

J

João Castro•7/29/24, 3:41 PM

Since I seem to have a similar problem, i'm also sending my IDs in case it helps

Queue: 26149965a00b4a65a047794b2a28b3f2
Account: 978f7d8fe0d7c314743586a46e505f14

J

João Castro•7/29/24, 3:43 PM

This happened while running some load tests. Some messages took 20+ minutes to be processed

Kkev-ac Don't quote me on that but it may be possible that event notifications don't wor...

P

Pranshu Maheshwari•7/29/24, 6:20 PM

that's correct! thanks for highlighting this, I'll add it to the docs

B

Ben•7/30/24, 12:41 AM

So I see that we can have 10,000 queues per account enabling easy sharding:

More queues per account - up to 10,000
Developers building on Queues can now create up to 10,000 queues per account, enabling easier per-user, per-job and sharding use-cases.

Refer to Limits to learn more about Queues’ current limits.

What is the recommendationf for doing this?

Is it a worker-per-tenant situation

Or a binding-per-tenant.

Are there any resources for these approaches or are they still in progress thanks!

PPranshu Maheshwari the limit is per queue. so you can manually shard / distribute messages across m...

J

josh•7/30/24, 7:59 PM

is there a rough ETA when we will see the higher limits?

Jjosh is there a rough ETA when we will see the higher limits? 😆

P

Pranshu Maheshwari•7/31/24, 6:56 PM

we're actively working on it

i'll share news soon once I have concrete dates

R

rbravo•8/1/24, 3:26 AM

Can I have multiple consumers in one Worker, each one processing different queues?

Context:

I'm reading this in the docs:

Best practice
Configure a single consumer per queue. This both logically separates your queues, and ensures that errors (failures) in processing messages from one queue do not impact your other queues.

J

josh•8/1/24, 9:13 AM

I believe you can only have one consumer per worker but that worker can process multiple queues

“Notably, you can use the same consumer with multiple queues. The queue handler that defines your consumer Worker will be invoked by the queues it is connected to.”

L

lupin•8/1/24, 10:22 AM

Hey team ! I hope you're doing great ?

Is there an API to see how many data are remaining in the queue ? I only found this:

- https://developers.cloudflare.com/queues/observability/metrics/

Should we use the Get average Queue backlog over time period and query for the last 5 minutes ?

Cheers !

Cloudflare Docs

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

PPranshu Maheshwari the limit is per queue. so you can manually shard / distribute messages across m...

S

Shivek•8/1/24, 2:05 PM

By manually sharding, you mean to have 2 or more queues for the same task?
Like NotificationsQueue0, NotificationsQueue1 and enque in a round-robin manner ?

T

Tobias•8/2/24, 10:34 AM

Is there any plans on making it so CloudFlare queues works when running wrangler dev --remote, if so what is the estimated timeframe for this?

Currently I am trying to setup a local development environment for running queues and CloudFlare Browser Rendering, but Browser Rendering requires --remote flag and Queues doesn't work with --remote flag. So I am kind of stuck in a limbo

TTobias Is there any plans on making it so CloudFlare queues works when running wrangler...

P

Pranshu Maheshwari•8/2/24, 2:32 PM

it's on our roadmap. we'll likely ship it in the next few months, but it's too soon to say exactly when

P

Ping for toast•8/4/24, 7:21 PM

How many fetch requests may I make from queues? I've found references to unbundled and bundled usage models, but these models are deprecated

PPing for toast How many fetch requests may I make from queues? I've found references to unbundl...

P

Pranshu Maheshwari•8/4/24, 9:29 PM

Queues consumers have the same limits as normal Workers (listed here: https://developers.cloudflare.com/workers/platform/limits/#subrequests)

Cloudflare Docs

Limits · Cloudflare Workers docs

Cloudflare Workers plan and platform limits.

PPranshu Maheshwari Queues consumers have the same limits as normal Workers (listed here: https://de...

P

Ping for toast•8/4/24, 9:30 PM

The limit for subrequests a Worker can make is 50 per request on the Bundled usage model or 1,000 per request on the Unbound usage model

Right, but aren't these models deprecated

P

Ping for toast•8/4/24, 9:30 PM

All users on the Workers Paid plan have been automatically migrated from the Bundled and Unbound usage models to the Standard usage model

P

Ping for toast•8/4/24, 9:31 PM

So I don't know what applies because I'm on the standard model

P

Ping for toast•8/4/24, 9:31 PM

ah, found it

PPing for toast So I don't know what applies because I'm on the standard model

P

Pranshu Maheshwari•8/4/24, 9:31 PM

standard / legacy paid plans have the same limits btw

P

Ping for toast•8/4/24, 9:31 PM

docs should be updated :)

PPranshu Maheshwari standard / legacy paid plans have the same limits btw 🙂

P

Ping for toast•8/4/24, 9:31 PM

cool, thanks

P

Ping for toast•8/4/24, 9:32 PM

thats a lot of requests, very exciting!!

M

mau•8/6/24, 8:51 AM

Getting weird issue with delays

const wait = getDelayInSeconds('14:12', 'Asia/Kolkata');
console.log(wait, 'wait');
await env.YOUR_QUEUE.send({ message: 'okay', date: new Date(), name: 'queue' }, { delaySeconds: wait });

const wait = getDelayInSeconds('14:12', 'Asia/Kolkata');
console.log(wait, 'wait');
await env.YOUR_QUEUE.send({ message: 'okay', date: new Date(), name: 'queue' }, { delaySeconds: wait });

const wait = getDelayInSeconds('14:12', 'Asia/Kolkata');
console.log(wait, 'wait');
await env.YOUR_QUEUE.send({ message: 'okay', date: new Date(), name: 'queue' }, { delaySeconds: wait });

const wait = getDelayInSeconds('14:12', 'Asia/Kolkata');
console.log(wait, 'wait');
await env.YOUR_QUEUE.send({ message: 'okay', date: new Date(), name: 'queue' }, { delaySeconds: wait });

wait has value of 85912

M

mau•8/6/24, 8:54 AM

error Error: Queue send failed: Internal Server Error
    at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6)
    at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10)
    at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)
[wrangler-UserWorker:err] TypeError: Invalid statusText
    at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10)
    at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14)
    at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

error Error: Queue send failed: Internal Server Error
    at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6)
    at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10)
    at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)
[wrangler-UserWorker:err] TypeError: Invalid statusText
    at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10)
    at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14)
    at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

error Error: Queue send failed: Internal Server Error
    at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6)
    at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10)
    at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)
[wrangler-UserWorker:err] TypeError: Invalid statusText
    at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10)
    at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14)
    at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

error Error: Queue send failed: Internal Server Error
    at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6)
    at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10)
    at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)
[wrangler-UserWorker:err] TypeError: Invalid statusText
    at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10)
    at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14)
    at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

M

mau•8/6/24, 8:57 AM

I am trying to build a notifications system that allows users to configure the time for notifications, whether it's set for the end of the day, weekly, or at another interval.

any suggestions on how to do that?

Mmau I am trying to build a notifications system that allows users to configure the t...

B

boop•8/6/24, 10:44 AM

Durable object alarms?
https://developers.cloudflare.com/durable-objects/api/alarms/

Cloudflare Docs

Alarms · Cloudflare Durable Objects docs

Durable Objects alarms allow you to schedule the Durable Object to be woken up at a time in the future. When the alarm’s scheduled time comes, the …

M

mau•8/6/24, 10:46 AM

I’m using D1, so not sure I can use alarms. Until unless I implement some code which stores in durable objects and wake up at certain time and then pushes the data to d1.

Mmau I’m using D1, so not sure I can use alarms. Until unless I implement some code w...

H

Hard@Work•8/6/24, 10:53 AM

Create a DO for each notification the user sets, then set an alarm

M

mau•8/6/24, 10:58 AM

Just to clarify… you mean one do object per notification and then set the alarm?

M

mau•8/6/24, 10:58 AM

Is there is any limit on alarm time limits?

Mmau Just to clarify… you mean one do object per notification and then set the alarm?

H

Hard@Work•8/6/24, 11:19 AM

That is correct. As for limits for the time, as long as it is representable, it should be able to fire

M

mau•8/6/24, 11:20 AM

I will give it a try. Thanks for the solution

D

Dlanda Griffin•8/8/24, 4:17 PM

D

dsds10•8/8/24, 6:11 PM

Can somebody explain to me why there is a generous limit of 10k queues per account? Maybe I‘m missing something obvious, but only pull is supported over HTTP, push would still need a worker binding (there is a max of 500 workers per account, although you can bind multiple queues to a single worker iirc). The only explanation would be workers for platforms, but then kv namespace and r2 bucket limits won‘t match that idea

Ddsds10 Can somebody explain to me why there is a generous limit of 10k queues per accou...

M

Matt Silverlock•8/8/24, 8:52 PM

Many folks have more than one script and Workers for Platforms use-cases. One Worker script can effectively bind ~5000 or so resources. Other products have different metadata constraints.

MMatt Silverlock Many folks have more than one script _and_ Workers for Platforms use-cases. One ...

D

dsds10•8/9/24, 12:07 PM

got it thx

F

flyingshoes•8/12/24, 8:42 PM

Is there a limit to how many tasks I can push into a queue? I'm thinking of having a cron job push tasks into a queue, and the limit could be pretty high (potentially millions). would the limit be the 1200 requests/ 5 minutes?

Fflyingshoes Is there a limit to how many tasks I can push into a queue? I'm thinking of havi...

J

John Spurlock•8/12/24, 8:42 PM

the limits are documented here: https://developers.cloudflare.com/queues/platform/limits/

Cloudflare Docs

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

KKam I'm seeing queue `send` and `sendBatch` calls taking anywhere from ~250ms to ove...

B

Brett Willis•8/12/24, 8:47 PM

Can't comment on those latencies myself, but you could consider using

ctx.waitUntil()

ctx.waitUntil()

ctx.waitUntil()

ctx.waitUntil() to more cheaply and easily defer those operations asynchronously without holding back the response.

The 400msg/s limit of a queue is bound to the single producer, correct? If I bind for example 10 que

Similar Threads

Similar Threads

Similar Threads