Cloudflare Developers•2y ago

is there a rough ETA when we will see the higher limits? 😆

is there a rough ETA when we will see the higher limits?

Jjosh is there a rough ETA when we will see the higher limits? 😆

Pranshu Maheshwari•7/31/24, 6:56 PM

we're actively working on it

i'll share news soon once I have concrete dates

rbravo•8/1/24, 3:26 AM

Can I have multiple consumers in one Worker, each one processing different queues?

Context:

I'm reading this in the docs:

Best practice
Configure a single consumer per queue. This both logically separates your queues, and ensures that errors (failures) in processing messages from one queue do not impact your other queues.

joshOP•8/1/24, 9:13 AM

I believe you can only have one consumer per worker but that worker can process multiple queues

“Notably, you can use the same consumer with multiple queues. The queue handler that defines your consumer Worker will be invoked by the queues it is connected to.”

lupin•8/1/24, 10:22 AM

Hey team ! I hope you're doing great ?

Is there an API to see how many data are remaining in the queue ? I only found this:

- https://developers.cloudflare.com/queues/observability/metrics/

Should we use the Get average Queue backlog over time period and query for the last 5 minutes ?

Cheers !

Cloudflare Docs

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

PPranshu Maheshwari the limit is per queue. so you can manually shard / distribute messages across m...

Shivek•8/1/24, 2:05 PM

By manually sharding, you mean to have 2 or more queues for the same task?
Like NotificationsQueue0, NotificationsQueue1 and enque in a round-robin manner ?

Tobias•8/2/24, 10:34 AM

Is there any plans on making it so CloudFlare queues works when running wrangler dev --remote, if so what is the estimated timeframe for this?

Currently I am trying to setup a local development environment for running queues and CloudFlare Browser Rendering, but Browser Rendering requires --remote flag and Queues doesn't work with --remote flag. So I am kind of stuck in a limbo

TTobias Is there any plans on making it so CloudFlare queues works when running wrangler...

Pranshu Maheshwari•8/2/24, 2:32 PM

it's on our roadmap. we'll likely ship it in the next few months, but it's too soon to say exactly when

Ping for toast•8/4/24, 7:21 PM

How many fetch requests may I make from queues? I've found references to unbundled and bundled usage models, but these models are deprecated

PPing for toast How many fetch requests may I make from queues? I've found references to unbundl...

Pranshu Maheshwari•8/4/24, 9:29 PM

Queues consumers have the same limits as normal Workers (listed here: https://developers.cloudflare.com/workers/platform/limits/#subrequests)

Cloudflare Docs

Limits · Cloudflare Workers docs

Cloudflare Workers plan and platform limits.

PPranshu Maheshwari Queues consumers have the same limits as normal Workers (listed here: https://de...

Ping for toast•8/4/24, 9:30 PM

The limit for subrequests a Worker can make is 50 per request on the Bundled usage model or 1,000 per request on the Unbound usage model

Right, but aren't these models deprecated

Ping for toast•8/4/24, 9:30 PM

All users on the Workers Paid plan have been automatically migrated from the Bundled and Unbound usage models to the Standard usage model

Ping for toast•8/4/24, 9:31 PM

So I don't know what applies because I'm on the standard model

Ping for toast•8/4/24, 9:31 PM

ah, found it

PPing for toast So I don't know what applies because I'm on the standard model

Pranshu Maheshwari•8/4/24, 9:31 PM

standard / legacy paid plans have the same limits btw

Ping for toast•8/4/24, 9:31 PM

docs should be updated :)

PPranshu Maheshwari standard / legacy paid plans have the same limits btw 🙂

Ping for toast•8/4/24, 9:31 PM

cool, thanks

Ping for toast•8/4/24, 9:32 PM

thats a lot of requests, very exciting!!

mau•8/6/24, 8:51 AM

Getting weird issue with delays

const wait = getDelayInSeconds('14:12', 'Asia/Kolkata');
console.log(wait, 'wait');
await env.YOUR_QUEUE.send({ message: 'okay', date: new Date(), name: 'queue' }, { delaySeconds: wait });

const wait = getDelayInSeconds('14:12', 'Asia/Kolkata');
console.log(wait, 'wait');
await env.YOUR_QUEUE.send({ message: 'okay', date: new Date(), name: 'queue' }, { delaySeconds: wait });

wait has value of 85912

mau•8/6/24, 8:54 AM

error Error: Queue send failed: Internal Server Error
    at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6)
    at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10)
    at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)
[wrangler-UserWorker:err] TypeError: Invalid statusText
    at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10)
    at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14)
    at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

error Error: Queue send failed: Internal Server Error
    at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6)
    at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10)
    at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10)
[wrangler-UserWorker:err] TypeError: Invalid statusText
    at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10)
    at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14)
    at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

mau•8/6/24, 8:57 AM

I am trying to build a notifications system that allows users to configure the time for notifications, whether it's set for the end of the day, weekly, or at another interval.

any suggestions on how to do that?

Mmau I am trying to build a notifications system that allows users to configure the t...

boop•8/6/24, 10:44 AM

Durable object alarms?
https://developers.cloudflare.com/durable-objects/api/alarms/

Cloudflare Docs

Alarms · Cloudflare Durable Objects docs

Durable Objects alarms allow you to schedule the Durable Object to be woken up at a time in the future. When the alarm’s scheduled time comes, the …

mau•8/6/24, 10:46 AM

I’m using D1, so not sure I can use alarms. Until unless I implement some code which stores in durable objects and wake up at certain time and then pushes the data to d1.

Mmau I’m using D1, so not sure I can use alarms. Until unless I implement some code w...

Hard@Work•8/6/24, 10:53 AM

Create a DO for each notification the user sets, then set an alarm

mau•8/6/24, 10:58 AM

Just to clarify… you mean one do object per notification and then set the alarm?

mau•8/6/24, 10:58 AM

Is there is any limit on alarm time limits?

Mmau Just to clarify… you mean one do object per notification and then set the alarm?

Hard@Work•8/6/24, 11:19 AM

That is correct. As for limits for the time, as long as it is representable, it should be able to fire

mau•8/6/24, 11:20 AM

I will give it a try. Thanks for the solution

Dlanda Griffin•8/8/24, 4:17 PM

dsds10•8/8/24, 6:11 PM

Can somebody explain to me why there is a generous limit of 10k queues per account? Maybe I‘m missing something obvious, but only pull is supported over HTTP, push would still need a worker binding (there is a max of 500 workers per account, although you can bind multiple queues to a single worker iirc). The only explanation would be workers for platforms, but then kv namespace and r2 bucket limits won‘t match that idea

Ddsds10 Can somebody explain to me why there is a generous limit of 10k queues per accou...

Matt Silverlock•8/8/24, 8:52 PM

Many folks have more than one script and Workers for Platforms use-cases. One Worker script can effectively bind ~5000 or so resources. Other products have different metadata constraints.

MMatt Silverlock Many folks have more than one script _and_ Workers for Platforms use-cases. One ...

dsds10•8/9/24, 12:07 PM

got it thx

flyingshoes•8/12/24, 8:42 PM

Is there a limit to how many tasks I can push into a queue? I'm thinking of having a cron job push tasks into a queue, and the limit could be pretty high (potentially millions). would the limit be the 1200 requests/ 5 minutes?

Fflyingshoes Is there a limit to how many tasks I can push into a queue? I'm thinking of havi...

John Spurlock•8/12/24, 8:42 PM

the limits are documented here: https://developers.cloudflare.com/queues/platform/limits/

Cloudflare Docs

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

KKam I'm seeing queue `send` and `sendBatch` calls taking anywhere from ~250ms to ove...

Brett Willis•8/12/24, 8:47 PM

Can't comment on those latencies myself, but you could consider using ctx.waitUntil()ctx.waitUntil() to more cheaply and easily defer those operations asynchronously without holding back the response.

JJohn Spurlock the limits are documented here: https://developers.cloudflare.com/queues/platfor...

flyingshoes•8/12/24, 8:58 PM

got it -- so 400 messages per second. thank you.

KKam I'm seeing queue `send` and `sendBatch` calls taking anywhere from ~250ms to ove...

Pranshu Maheshwari•8/12/24, 10:13 PM

it's a bit higher than expected, but not out of the ordinary. We're working on improvements that should improve latency for sending messages to a queue. Those improvements should ship by Oct.

KKam Thanks for the info Pranshu! Any idea what latency you are targeting with the im...

Pranshu Maheshwari•8/13/24, 3:11 PM

It's too soon to share specific targets, but stay tuned!

NoGameNoFun•8/14/24, 3:26 AM

Hi everyone, I like the worker and queue features in cloudflare, and been trying out about few weeks already.

I got a question, I noticed the consumer for the queue, after some time (like 2 days), it will stop processing the queue message. (the backlog size increase and doesnt get consume)

I attached the screenshot on the consumer setting, is this a bug or is just because of my consumer setting?

jordanfinners•8/14/24, 9:13 PM

Please can I provide some feedback/feature requests for cloudflare queues

Firstly the dev experience inside workers is pretty great! It would be awesome if it could work with Pages functions so I don't have to deploy separate consumers but thats a nit really.

The metrics in the dashboard are useful but its missing the main one for me, how many messages are in the queue at that point in time, especially important for deadletters
Redriving of deadletter messages, either a button in the UI or a way of doing it via HTTP or JS.
Currently you can pull the messages from the queue via HTTP but can't acknowledge them or push back into the queue locally, meaning I would need to deploy a worker just to redrive each queue which seems excessive and I need to control what I'm redriving or not.
I had to do 200 messages manually this morning
Alerts on the queue length, ie. If messages drop into a deadletter
Would be helpful if the error message was attached to the queue message in some way like SQS does
Being able to send multiple messages via the UI either file upload or pasting in
Documentation is missing for the Preview endpoint in the Queue REST API docs
Its only happened once but I've had one message not get sent, even though it said it has, which worries me slightly.
If possible increase the batch size, up to 200 would be helpful
Alongside this if possible to increase the timeout period as well, I'm running async reports and these can take several minutes to process and so they are timing out/hitting worker limits.

Happy to talk this through or provide more info if

Wwindygod9986 Does anyone know if cloudflare has any plans to support queue consumers on page...

Matt Silverlock•8/17/24, 12:38 AM

Not in the near term. Pages doesn’t have the concept of any non-request handler (cron, queues, tail)

lulupineapple•8/18/24, 2:06 PM

Hi, how do I deal with queue consumer timeout issue? i.e. make sure queue consumer completed the task with payload? Is there anyway to have queue retry calling consumer unless consumer explicitly acknowledge the payload was processed susessfully?

oof2win2•8/21/24, 12:21 PM

how can i set proper types for a queue?

oof2win2•8/21/24, 12:23 PM

like it seems that i need to change it on the generated env types but then i can't have them be generated

error Error: Queue send failed: Internal Server Error at async Object.handler (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/src/index.ts:34:6) at async jsonError (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-miniflare3-json-error.ts:22:10) at async drainBody (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/wrangler/templates/middleware/middleware-ensure-req-body-drained.ts:5:10) [wrangler-UserWorker:err] TypeError: Invalid statusText at HttpError.toResponse (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:16:10) at QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/router.worker.ts:50:14) at async QueueBrokerObject.fetch (file:///Users/nicolethomas/Desktop/NTML/HomeworkHelp/throbbing-salad-88dd/node_modules/miniflare/src/workers/shared/object.worker.ts:148:11)

is there a rough ETA when we will see the higher limits? 😆

Similar Threads