Hi! Can someone from Cloudflare help to raise my API limits? I constantly run into 429 when polling

Hi! Can someone from Cloudflare help to raise my API limits? I constantly run into 429 when polling queues with a larger fleet of EC2 instances.

FFabian We use queues to aggregate calls to an external API, only thing that would make ...

telliks•6/27/24, 6:37 PM

I second this. GCP PubSub has this concept to provide an “ordering key”. Messages of the same ordering key needs to be processed in a specific order without delaying others.

MMichael Wittig Hi! Can someone from Cloudflare help to raise my API limits? I constantly run in...

Pranshu Maheshwari•6/27/24, 9:23 PM

Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVOboH9SlutMonzbhCxuuuOmiU1L_I5O2CFbXf_XXMRg/viewform

Google Docs

Cloudflare Developer Platform: Limit Increase Request Form

Want to increase the limits of your current plan? Let us know what your requirements are here.

PPranshu Maheshwari Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVObo...

Michael WittigOP•6/28/24, 7:23 AM

Thanks @Pranshu Maheshwari I tried that 1 month ago with no response. I just submitted the form again, let's see what happens

johtso•6/28/24, 6:39 PM

How do I reject a message?

Jjohtso How do I reject a message?

Hard@Work•6/28/24, 6:41 PM

Like, so that it doesn't get processed again?

johtso•6/28/24, 6:41 PM

if I message.ack()message.ack() and then await env.<MY_DEAD_LETTER_QUEUE>.send(foo)await env.<MY_DEAD_LETTER_QUEUE>.send(foo) is it guaranteed that either both will succeed or neither will?

johtso•6/28/24, 6:42 PM

basically if I process a message and know it can't succeed to I want to send it straight to my naughty bin

Jjohtso if I `message.ack()` and then `await env.<MY_DEAD_LETTER_QUEUE>.send(foo)` is it...

Hard@Work•6/28/24, 6:42 PM

Maybe the other way around?

Hard@Work•6/28/24, 6:42 PM

Push to the DLQ, then ack() if no errors are hit?

johtso•6/28/24, 6:43 PM

hmm

HHard@Work Push to the DLQ, then ack() if no errors are hit?

johtso•6/28/24, 6:44 PM

I guess there's a chance that acking fails and the message gets retried and I end up putting it on my failure queue twice

johtso•6/28/24, 6:44 PM

not the end of the world I guess

johtso•6/28/24, 6:44 PM

just feels like there should be an atomic way of routing a message to another queue though..

johtso•6/28/24, 6:45 PM

not sure if that's something that exists with other queue systems.. it's been a while since I used them

johtso•6/28/24, 6:47 PM

I think rabbitmq for example might have a concept of rejecting a message with requeue=falserequeue=false so it goes straight to the dead letter queue

johtso•6/28/24, 6:47 PM

seems like a reasonable feature

Jjohtso I guess there's a chance that acking fails and the message gets retried and I en...

Hello, I’m Allie!•6/28/24, 6:51 PM

I don’t think that can fail, or at least it should be so unlikely that you shouldn’t need to worry

HHello, I’m Allie!I don’t *think* that can fail, or at least it should be so unlikely that you sho...

Matt Silverlock•6/28/24, 9:05 PM

If it fails you will get an exception, so you will know it failed.

MMatt Silverlock If it fails you will get an exception, so you will _know_ it failed.

Hello, I’m Allie!•6/28/24, 9:48 PM

If an ack fails, should it be retried in the same block, or would it have failed for the entire batch run?

paddebal•6/28/24, 10:10 PM

did someone really tag everyone here

paddebal•6/28/24, 10:10 PM

smh

HHello, I’m Allie!If an ack fails, should it be retried in the same block, or would it have failed...

Matt Silverlock•6/28/24, 11:11 PM

It’s possible for a single ack() to fail (rare as it may be)

Milo•7/1/24, 1:34 PM

Regarding the Queues REST API, I'm looking at the docs for the 'Pull Queue Messages' endpoint and documented there is the result_inforesult_info field.

This field would be quite useful for me because I'd like to see the queue backlog size, but it isn't actually returned by the API. I noticed it doesn't have RequiredRequired next to it, but is there a way to enable returning that field or does it just not exist?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

MMilo Regarding the Queues REST API, I'm looking at [the docs for the 'Pull Queue Mess...

Matt Silverlock•7/1/24, 1:59 PM

result_inforesult_info is for paginated results - not for backlogs. We don’t currently return the backlog count in the response at all.

Milo•7/1/24, 2:00 PM

Thanks

Ben•7/1/24, 2:49 PM

Hey for some reason my deployed queues are simply not consuming the queues.

I get no logs, no errors, they just dont process, any tips for debugging this?

actually I think I think I fixed it xD

rewbs•7/1/24, 2:53 PM

Hi all, I'm working with an R2 event notification queue and everything is working smoothly, love the feature and ease of use. I was wondering if there's any chance object metadata might be included in the upload notification event payload in the future?

PPranshu Maheshwari Thanks for highlighting this, we're looking into it

carlos-alberto•7/1/24, 8:10 PM

Hey @Pranshu Maheshwari , was there any update on our issue?

Ccarlos-alberto Hey @Pranshu Maheshwari , was there any update on our issue?

Pranshu Maheshwari•7/1/24, 10:19 PM

We noticed networking issues with a few queues last week. We're still investigating the root cause, but some of the queues that were returning 500s are working fine now. It's one of the issues we're ironing out while queues is in beta!

Shivek•7/2/24, 11:06 AM

Hey, is there no way to work with queues locally?

Shivek•7/2/24, 11:06 AM

I'm building an app using KV and Queues. I have set remote=true so I can read env variables from dev workers env, and use dev kv bucket.

Shivek•7/2/24, 11:07 AM

But when I introduced Queues, I got this warning and my app breaks down:

▲ [WARNING] Queues are currently in Beta and are not supported in wrangler dev remote mode.

▲ [WARNING] Queues are currently in Beta and are not supported in wrangler dev remote mode.

SShivek Hey, is there no way to work with queues locally?

Hard@Work•7/2/24, 11:24 AM

Sort of. As you noted, you can use them locally, but then you can't connect to any remote resources

HHard@Work Sort of. As you noted, you can use them locally, but then you can't connect to a...

Shivek•7/2/24, 11:36 AM

So I turned off remote=true and injected my env variables as .dev.vars. But even then I'm not able to use queues.

Shivek•7/2/24, 11:37 AM

  try {
    console.log(1);
    const q = queue.factory(c);
    console.log(1.2);
    await q.send(item);
    console.log(2);
  } catch (error) {
    console.log(3);
    console.log(error);
  }

  try {
    console.log(1);
    const q = queue.factory(c);
    console.log(1.2);
    await q.send(item);
    console.log(2);
  } catch (error) {
    console.log(3);
    console.log(error);
  }

Some sample code.

q.send

q.send

fails silently. 2 is never printed and I get no errors.

SShivek ```ts try { console.log(1); const q = queue.factory(c); console.lo...

Hard@Work•7/2/24, 11:37 AM

Both the Producer and Consumer must be the same Worker

Hard@Work•7/2/24, 11:37 AM

At least for now

Hard@Work•7/2/24, 11:38 AM

(iirc)

HHard@Work Both the Producer and Consumer must be the same Worker

Shivek•7/2/24, 11:39 AM

I have no consumer setup right now. I'm just trying to see if send is working

Shivek•7/2/24, 11:39 AM

it is not at the moment

SShivek I have no consumer setup right now. I'm just trying to see if send is working

Hard@Work•7/2/24, 11:39 AM

Yeah, it might fail if there is nothing to write to, I'm not sure

HHard@Work Yeah, it *might* fail if there is nothing to write to, I'm not sure

Shivek•7/2/24, 11:45 AM

My config seems correct and I see the resource being attached when I run wrangler dev

Shivek•7/2/24, 11:45 AM

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

SShivek ```Your worker has access to the following bindings: - KV Namespaces: - xxxx: ...

Hard@Work•7/2/24, 11:45 AM

Yes, but if there isn't a consumer for that queue, I would guess it might still fail

Shivek•7/2/24, 11:46 AM

okay, so both producer and consumers need to be defined before I can write?

SShivek okay, so both producer and consumers need to be defined before I can write?

Hard@Work•7/2/24, 11:47 AM

Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently failing. If you are able to wait a bit, I may be able to test it out and see if there is something else going wrong

HHard@Work Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently faili...

Shivek•7/2/24, 11:48 AM

I can wait, I'll try to set a consumer in the meanwhile

Hi! Can someone from Cloudflare help to raise my API limits? I constantly run into 429 when polling

Similar Threads

Similar Threads

Similar Threads