Since ~18:00 yesterday one of our queues has been erroring with `Queue send failed: Internal Server

Since ~18:00 yesterday one of our queues has been erroring with

Queue send failed: Internal Server Error

Queue send failed: Internal Server Error

Queue send failed: Internal Server Error

Queue send failed: Internal Server Error.

Trying to redeploy the worker gives

workers.api.error.unknown [code: 10013]

workers.api.error.unknown [code: 10013]

workers.api.error.unknown [code: 10013]

workers.api.error.unknown [code: 10013], as does trying to delete the queue. Anyone able to help figure out what is going on here?

Queue ID: 18dbd75d6a1946b48f614689ed23bec9

Ccarlos-alberto Since ~18:00 yesterday one of our queues has been erroring with `Queue send fail...

Matt Silverlock•6/26/24, 10:27 AM

Which timezone and what is your account ID? (safe to post here)

MMatt Silverlock Which timezone and what is your account ID? (safe to post here)

carlos-albertoOP•6/26/24, 11:02 AM

Timezone: BST
Account ID: ef98901aa25fef025cee6651b601b491

Ccarlos-alberto Timezone: BST Account ID: ef98901aa25fef025cee6651b601b491

Matt Silverlock•6/26/24, 12:27 PM

Are you still seeing issues? (either sending messages or creating a Queue)

Was this sporadic?

MMatt Silverlock Are you still seeing issues? (either sending messages or creating a Queue) Was...

carlos-albertoOP•6/26/24, 12:43 PM

This is ongoing, we are unable to interact with that queue in any way. Even trying to update the "Delivery delay" gives the same error

Ccarlos-alberto This is ongoing, we are unable to interact with that queue in any way. Even tryi...

Matt Silverlock•6/26/24, 12:44 PM

@Pranshu Maheshwari ^

carlos-albertoOP•6/26/24, 12:44 PM

We've deployed a new version of our worker under a new name, and a new queue so we can work around this, but it's ongoing for this queue

Pranshu Maheshwari•6/26/24, 1:27 PM

Thanks for highlighting this, we're looking into it

Original message was deleted

santosh•6/26/24, 4:58 PM

@FraznoFire ?

PPranshu Maheshwari Hi @everyone! Pranshu here. I’m a product manager at Cloudflare and I’ve recentl...

Fabian•6/26/24, 5:25 PM

We use queues to aggregate calls to an external API, only thing that would make queues for that use-case better is that batches currently cannot be grouped by a (custom) identifier - say we scope all the data to a single customer and group it under a 30s queue timeframe, we could have more efficient calls to the external API (since they also operate under a per-customer/ID resource scope).
This is relevant because the external API has relatively strict account rate-limits (not scoped to the ID).
Right now it's hit or miss since possibly 100 different customers could join the same batch effectively resulting in groups of 1 and thus 100 API calls.
In this case we would prefer grouped batching over quick execution of a random batch if you understand what I mean

andyy•6/26/24, 5:49 PM

is there any kind of queue bindings-per-worker limit? (sorry if I missed it anywhere in the docs). curious whether it's possible/sensible to have a queue-per-customer architecture all written to by the same producer worker. thanks!

Ttizmagik Hey folks. On latest wrangler (`3.62.0`) and it looks like trying to do `wrangle...

Pranshu Maheshwari•6/26/24, 8:32 PM

Hmm I haven't been able to replicate this on my side. CLI arguments should override what you've specified in

wrangler.toml

wrangler.toml

Thisjrodriguez•6/27/24, 6:28 AM

Hello,

I intend to use queues to save the data of each visit and then import them to have an analytic.

Is this the use of queues? I understand that yes, but if someone confirms me I would appreciate it.

TThisjrodriguez Hello, I intend to use queues to save the data of each visit and then import th...

Matt Silverlock•6/27/24, 11:19 AM

What kind of analytics? Writing directly to WAE may be more effective: https://developers.cloudflare.com/analytics/analytics-engine/

Cloudflare Docs

Workers Analytics Engine · Cloudflare Analytics docs

Workers Analytics Engine provides unlimited-cardinality analytics at scale, via a built-in API to write data points from Workers, and a SQL API to …

MMatt Silverlock What kind of analytics? Writing directly to WAE may be more effective: https://d...

Thisjrodriguez•6/27/24, 11:47 AM

a personalized one, I just wanted to know if it is useful for what I proposed.

TThisjrodriguez a personalized one, I just wanted to know if it is useful for what I proposed.

Matt Silverlock•6/27/24, 11:50 AM

It’s hard to say: you haven’t provided enough detail.

A Queue simply puts messages onto a queue for you to consume later.

That can be useful for deferring work or batching tasks together for efficiency, and/or getting “slow” API calls off the hot path so they don’t slow down user-facing responses.

andyy•6/27/24, 1:39 PM

Hi again! Is it possible to test pull consumers locally? I.e. fetch() from some wrangler local dev host rather than api.cloudflare.com/.../queues/../messages/pull. Big thanks!

Michael Wittig•6/27/24, 6:01 PM

Hi! Can someone from Cloudflare help to raise my API limits? I constantly run into 429 when polling queues with a larger fleet of EC2 instances.

FFabian We use queues to aggregate calls to an external API, only thing that would make ...

telliks•6/27/24, 6:37 PM

I second this. GCP PubSub has this concept to provide an “ordering key”. Messages of the same ordering key needs to be processed in a specific order without delaying others.

MMichael Wittig Hi! Can someone from Cloudflare help to raise my API limits? I constantly run in...

Pranshu Maheshwari•6/27/24, 9:23 PM

Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVOboH9SlutMonzbhCxuuuOmiU1L_I5O2CFbXf_XXMRg/viewform

Google Docs

Cloudflare Developer Platform: Limit Increase Request Form

Want to increase the limits of your current plan? Let us know what your requirements are here.

PPranshu Maheshwari Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVObo...

Michael Wittig•6/28/24, 7:23 AM

Thanks @Pranshu Maheshwari I tried that 1 month ago with no response. I just submitted the form again, let's see what happens

johtso•6/28/24, 6:39 PM

How do I reject a message?

Jjohtso How do I reject a message?

Hard@Work•6/28/24, 6:41 PM

Like, so that it doesn't get processed again?

johtso•6/28/24, 6:41 PM

if I

message.ack()

message.ack()

and then

await env.<MY_DEAD_LETTER_QUEUE>.send(foo)

await env.<MY_DEAD_LETTER_QUEUE>.send(foo)

is it guaranteed that either both will succeed or neither will?

johtso•6/28/24, 6:42 PM

basically if I process a message and know it can't succeed to I want to send it straight to my naughty bin

Jjohtso if I `message.ack()` and then `await env.<MY_DEAD_LETTER_QUEUE>.send(foo)` is it...

Hard@Work•6/28/24, 6:42 PM

Maybe the other way around?

Hard@Work•6/28/24, 6:42 PM

Push to the DLQ, then ack() if no errors are hit?

johtso•6/28/24, 6:43 PM

hmm

HHard@Work Push to the DLQ, then ack() if no errors are hit?

johtso•6/28/24, 6:44 PM

I guess there's a chance that acking fails and the message gets retried and I end up putting it on my failure queue twice

johtso•6/28/24, 6:44 PM

not the end of the world I guess

johtso•6/28/24, 6:44 PM

just feels like there should be an atomic way of routing a message to another queue though..

johtso•6/28/24, 6:45 PM

not sure if that's something that exists with other queue systems.. it's been a while since I used them

johtso•6/28/24, 6:47 PM

I think rabbitmq for example might have a concept of rejecting a message with

requeue=false

requeue=false

so it goes straight to the dead letter queue

johtso•6/28/24, 6:47 PM

seems like a reasonable feature

Jjohtso I guess there's a chance that acking fails and the message gets retried and I en...

Hello, I’m Allie!•6/28/24, 6:51 PM

I don’t think that can fail, or at least it should be so unlikely that you shouldn’t need to worry

HHello, I’m Allie!I don’t *think* that can fail, or at least it should be so unlikely that you sho...

Matt Silverlock•6/28/24, 9:05 PM

If it fails you will get an exception, so you will know it failed.

MMatt Silverlock If it fails you will get an exception, so you will _know_ it failed.

Hello, I’m Allie!•6/28/24, 9:48 PM

If an ack fails, should it be retried in the same block, or would it have failed for the entire batch run?

paddebal•6/28/24, 10:10 PM

did someone really tag everyone here

paddebal•6/28/24, 10:10 PM

smh

HHello, I’m Allie!If an ack fails, should it be retried in the same block, or would it have failed...

Matt Silverlock•6/28/24, 11:11 PM

It’s possible for a single ack() to fail (rare as it may be)

Milo•7/1/24, 1:34 PM

Regarding the Queues REST API, I'm looking at the docs for the 'Pull Queue Messages' endpoint and documented there is the

result_info

result_info

field.

This field would be quite useful for me because I'd like to see the queue backlog size, but it isn't actually returned by the API. I noticed it doesn't have

Required

Required

next to it, but is there a way to enable returning that field or does it just not exist?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

MMilo Regarding the Queues REST API, I'm looking at [the docs for the 'Pull Queue Mess...

Matt Silverlock•7/1/24, 1:59 PM

result_info

result_info

is for paginated results - not for backlogs. We don’t currently return the backlog count in the response at all.

Milo•7/1/24, 2:00 PM

Thanks

Ben•7/1/24, 2:49 PM

Hey for some reason my deployed queues are simply not consuming the queues.

I get no logs, no errors, they just dont process, any tips for debugging this?

actually I think I think I fixed it xD

rewbs•7/1/24, 2:53 PM

Hi all, I'm working with an R2 event notification queue and everything is working smoothly, love the feature and ease of use. I was wondering if there's any chance object metadata might be included in the upload notification event payload in the future?

PPranshu Maheshwari Thanks for highlighting this, we're looking into it

carlos-albertoOP•7/1/24, 8:10 PM

Hey @Pranshu Maheshwari , was there any update on our issue?

Since ~18:00 yesterday one of our queues has been erroring with `Queue send failed: Internal Server

Similar Threads

Similar Threads

Similar Threads