hmm, I'm stumped, getting this 403 error message from the api when trying to pull messages from a pa

hmm, I'm stumped, getting this 403 error message from the api when trying to pull messages from a particular queue:

messages cannot be pulled unless http_pull mode is enabled

messages cannot be pulled unless http_pull mode is enabled

messages cannot be pulled unless http_pull mode is enabled

messages cannot be pulled unless http_pull mode is enabled

the queue has a single consumer of type

http_pull

http_pull

http_pull

http_pull, which I thought was the key to defining "http_pull mode":

{
  "queue_id": "8c720189e5c54eefa03f53eefb7162f1",
  "queue_name": "queue2-ci-dlq",
  "settings": {},
  "created_on": "2024-03-10T18:25:36.141403Z",
  "modified_on": "2024-03-10T18:25:36.141403Z",
  "producers_total_count": 0,
  "producers": [],
  "consumers_total_count": 1,
  "consumers": [
    {
      "settings": {
        "batch_size": 10,
        "max_retries": 100,
        "visibility_timeout_ms": 30000,
        "retry_delay": 0
      },
      "consumer_id": "d2a3644fd6cc40fab4e023ce65c451dc",
      "type": "http_pull"
    }
  ]
}

{
  "queue_id": "8c720189e5c54eefa03f53eefb7162f1",
  "queue_name": "queue2-ci-dlq",
  "settings": {},
  "created_on": "2024-03-10T18:25:36.141403Z",
  "modified_on": "2024-03-10T18:25:36.141403Z",
  "producers_total_count": 0,
  "producers": [],
  "consumers_total_count": 1,
  "consumers": [
    {
      "settings": {
        "batch_size": 10,
        "max_retries": 100,
        "visibility_timeout_ms": 30000,
        "retry_delay": 0
      },
      "consumer_id": "d2a3644fd6cc40fab4e023ce65c451dc",
      "type": "http_pull"
    }
  ]
}

{
  "queue_id": "8c720189e5c54eefa03f53eefb7162f1",
  "queue_name": "queue2-ci-dlq",
  "settings": {},
  "created_on": "2024-03-10T18:25:36.141403Z",
  "modified_on": "2024-03-10T18:25:36.141403Z",
  "producers_total_count": 0,
  "producers": [],
  "consumers_total_count": 1,
  "consumers": [
    {
      "settings": {
        "batch_size": 10,
        "max_retries": 100,
        "visibility_timeout_ms": 30000,
        "retry_delay": 0
      },
      "consumer_id": "d2a3644fd6cc40fab4e023ce65c451dc",
      "type": "http_pull"
    }
  ]
}

{
  "queue_id": "8c720189e5c54eefa03f53eefb7162f1",
  "queue_name": "queue2-ci-dlq",
  "settings": {},
  "created_on": "2024-03-10T18:25:36.141403Z",
  "modified_on": "2024-03-10T18:25:36.141403Z",
  "producers_total_count": 0,
  "producers": [],
  "consumers_total_count": 1,
  "consumers": [
    {
      "settings": {
        "batch_size": 10,
        "max_retries": 100,
        "visibility_timeout_ms": 30000,
        "retry_delay": 0
      },
      "consumer_id": "d2a3644fd6cc40fab4e023ce65c451dc",
      "type": "http_pull"
    }
  ]
}

tried removing/re-adding the pull consumer, double-checked the token has edit-queue permission, anyone else seen this before?

JJohn Spurlock hmm, I'm stumped, getting this 403 error message from the api when trying to pul...

ajgeiss0702•1/12/25, 4:17 PM

Have you tried enabling it via wrangler?

https://developers.cloudflare.com/queues/configuration/pull-consumers/#wrangler-cli

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A pull-based consumer can be useful when your message consumption rate is limited by upstream infrastructure or long-running tasks.

Aajgeiss0702 Have you tried enabling it via wrangler? https://developers.cloudflare.com/queu...

John SpurlockOP•1/12/25, 7:10 PM

yes, and through the dashboard - the config seems correct, it's similar to a prod dlq queue in that same account which works fine - it's just that queue 8c720189e5c54eefa03f53eefb7162f18c720189e5c54eefa03f53eefb7162f1 that gives

messages cannot be pulled unless http_pull mode is enabled

messages cannot be pulled unless http_pull mode is enabled

. I even tried previewing and deleting the 3 messages in there, didn't help

xintercept•1/15/25, 12:37 AM

anyone else run into problems with queues today? we hit a bunch of "Queue send failed: Service Temporarily Unavailable" around 2025-01-14T19:51:06.333Z

Xxintercept anyone else run into problems with queues today? we hit a bunch of "Queue send f...

xintercept•1/15/25, 12:38 AM

(doesn't look like anything was reported on https://www.cloudflarestatus.com/ related to Qs either)

Cloudflare Status

Welcome to Cloudflare's home for real-time and historical data on system performance.

chhatra_•1/15/25, 6:52 AM

Hello , I am building some application with cloudflare queues and worker . My consumer worker code checks if the system is free then it sends message from the queue to the system else the queue retries after cetain time 1200s .the queue retries the message but after delayed time has passed messages , the message is not processed from the queue but when I make a new request to the queue then the new message and previous message is processed ?

chhatra_•1/15/25, 6:53 AM

How to fix this ? what could be the causes for it . I want the consumer to automatically process the message after the retry time has passed

chhatra_•1/15/25, 7:04 AM

Does anyone know what settings should check ?

chhatra_•1/16/25, 3:42 AM

Actually , the queue was working fine previously . I noticed multiple repeatative requests from diffrent user agent so I applied code to filter that out. After that this happen, I commented out the code that filter it out on the basis of user agent , still it got the same problem. I had lost my original source code when my laptop broke down. I noticed to many requests on my queue so I made changes using the GUI interface provided by cloudflare. Directly modifying the compiled code . Can not using wrangler cli be a cause of this problem ?

Cchhatra_Hello , I am building some application with cloudflare queues and worker . My co...

Rakhim•1/16/25, 12:14 PM

Have you considered using Workflows instead? It feels better suited for retries/fallback scenarios, you can define the retry and backoff policy per workflow, and it can also sleep for long periods of time.

kcbrewron•1/17/25, 4:51 PM

Not sure this is the right channel but nothing for notifications that I saw. I would love to see the ability to configure notifications when my dlq reaches a certain threshold without having to query the metrics API constantly

allen•1/18/25, 11:49 AM

Hi
I'm using Cloudflare Workers and I'm passing messages between them using Queues. Within a batch of messages the code iterates over the messages in a for loop and after processing each message it explicitly acks the message, eg message.ack(). Despite this the message gets redelivered and the queue fills up with messages until they run out of retries.

ack'ing the batch doesn't work either (batch.ackAll()) and not acking at all and just returning also has no effect. So for now I've set the queus to have max_retries = 0 and no DLQ. Is there something obvious maybe that I've missed?

I'm going to head out now to try and enjoy the rest of my Saturday

sadel•1/18/25, 5:45 PM

would be amazing if you guys could add an extra configuration to queues for max dispatches / second so that we can fully wrap 3rd party apis and stay within rate limits

Cchhatra_Hello , I am building some application with cloudflare queues and worker . My co...

sup filistine•1/19/25, 11:56 AM

i saw this same behavior yesterday. curious what’s up. i didn’t pass an explicit retry delaySeconds for the message, just an empty .retry(), and was hoping it will respect the retry_delay configured under the wrangler consumer section. will do some more tests today

Skrrrrrrrrr•1/19/25, 7:55 PM

is there bug with the Turkey server?

TetraEtc•1/23/25, 12:55 AM

hey all, I've seen this before but can't find them right now.I'm looking for a good example repo for the following setup:
1. Publisher worker that recieves API request with data, and publishes the data to a queue. (Similar to this: https://developers.cloudflare.com/queues/examples/publish-to-a-queue-over-http/)
2. Consumer worker that takes individual (not batch) messages, and performs logic with them (specifically doing some API calls)

I'm also wanting it to retry the consumer twice, before sending to a dead letter queue that does stuff with the message data.

Brett Willis•1/23/25, 5:25 AM

I've had to disconnect the queue consumer in my application to effectively pause a certain processing pipeline while I sort out a bug.
As the messages accumulate in the queue without a consumer, will they safely stay around until I resume (until the 96hr retention period)?

rivage•1/23/25, 2:00 PM

Hey there, my Queue has had a concurrency of 3 for the last 12 hours. Failure rate is low. Maximum concurrency is 250. Any ideas on how to increase this?

pat•1/26/25, 12:12 PM

Hey all, wondering if theres a way to set a limit on the number of requests to a worker over a given period of time before it starts rejecting new requests? If I can avoid implementing it myself that'd be great

flof.fly•1/26/25, 8:22 PM

How do I increase the max concurrency workers to 500 (the maximum) when polling messages from Queue?

Rahul gangotri•1/27/25, 8:19 PM

I am running a browser rendering queue consumer which takes data and start the crawling of the mentioned link

If i start dev env, it says browser does not work in dev

And in remote queue is not working

So I decided to deploy but now there is an error and i am not able to get my console logs and any log on dashboard of worker also

Log page of worker just crashes constantly

Simon Colby•1/29/25, 4:36 PM

Hey guys - it's my first time building backend with cloudflare workers and I'm having a hard time with understanding queues behaviour. As you can see on my screenshot the deliverability is tragic, but the service in my mobile app works. What can be a reason?

SSimon Colby Hey guys - it's my first time building backend with cloudflare workers and I'm h...

Matt Silverlock•1/30/25, 12:54 PM

Messages are delivered to the DLQ when they reach the configured retry limit for the consumer. Without a DLQ configured, messages that reach the retry limit are deleted permanently.

https://developers.cloudflare.com/queues/configuration/dead-letter-queues/

your consumer is likely throwing exceptions over and over. Have you checked the logs?

Cloudflare Docs

Dead Letter Queues · Cloudflare Queues

A Dead Letter Queue (DLQ) is a common concept in a messaging system, and represents where messages are sent when a delivery failure occurs with a consumer after max_retries is reached. A Dead Letter Queue is like any other queue, and can be produced to and consumed from independently.

Mitya•1/30/25, 2:55 PM

Hi - I'm new to queues. Just made one, testing locally with --remote--remote, and got:

Queues are currently in Beta and are not supported in wrangler dev remote mode.

Given this, is there no way for me to test in development? I need --remote--remote for various other reasons; testing without that flag isn't an option.

MMitya Hi - I'm new to queues. Just made one, testing locally with `--remote`, and got:...

Stefan Stührmann•1/30/25, 4:05 PM

I'm halfway finished with the implementation of local queues using durable objects in miniflare. This will allow you to have local persistent queues that can be shared between multiple workers instead of the in memory array based queues and therefore you can test distributed / microservice architectures using queues fully locally. May I ask why you need --remote? For me the biggest issue is the lack of local queues between multiple workers (the main purpose of queues).

SStefan Stührmann I'm halfway finished with the implementation of local queues using durable objec...

Mitya•1/30/25, 5:32 PM

Thanks. I need remote for two reasons, mainly. Firstly, and for some unknown reason, my Worker refuses to connect to my remote PG DB (even though I can connect to it on my machine via terminal). The only workaround was to use the remote flag and access via a hyperdrive. This seems to be a problem unique to me and my DNS or network or whatever. Secondly, I use CF AI.

Mitya•1/30/25, 5:33 PM

Is there a preferred way to simulate queues locally with the remote flag? Sort of like the __schedule__schedule query string arg for testing CRON schedules locally.

Mitya•1/31/25, 10:49 PM

Question. The docs say consumers get 30 seconds of CPU time. I'm trying to understand what this means practically. If my consumer is sent a batch of 10 messages, and each of those messages needs sending to e.g. an AI analysis API, is that likely to max out the 30 seconds of CPU time? How can one reliably predict CPU time for a given task?

MMitya Question. The docs [say](https://developers.cloudflare.com/queues/platform/limit...

Isaac McFadyen•1/31/25, 10:52 PM

?cpu-time

SuperHelpflare•1/31/25, 10:52 PM

CPU Time: the amount of work the CPU needs to do, measured in msms. More CPU-heavy things will increase it (for example, hashing passwords). This is not the same as the time a

fetch

fetch

request takes.
Wall Time: the actual amount of time a Worker runs for, as though you measured it with a clock or your browser DevTools

Isaac McFadyen•1/31/25, 10:52 PM

CPU time is not wall time.

Isaac McFadyen•1/31/25, 10:52 PM

If you're making external requests then you're unlikely to get even near 30s.

Isaac McFadyen•1/31/25, 10:52 PM

More likely you'd be in the tens of ms total.

Rahul gangotri•2/1/25, 6:29 AM

Hi
I have deployed a queue that does the website crawling and have added many console logs

problem is that browser rendering only works with

 wrangler dev --remote

 wrangler dev --remote

but queue doesn't support it

if i am deploying its not showing my console, how to test it?

Rahul gangotri•2/1/25, 6:30 AM

i want to reuse browser session so i think queue is the best way, or can i use workflow also?

Rahul gangotri•2/1/25, 6:30 AM

i am crawling in a nested way to fetch the content of all provided links

RRahul gangotri Hi I have deployed a queue that does the website crawling and have added many co...

Hello, I’m Allie!•2/1/25, 6:33 AM

You can deploy it

Hello, I’m Allie!•2/1/25, 6:33 AM

That’s about it when you have a conflict like that

Rahul gangotri•2/1/25, 6:39 AM

If i deploy, how do i check my console logs for debugging?

Rahul gangotri•2/1/25, 6:40 AM

Logs page of worker does not shows console and breaks all the time

Rahul gangotri•2/1/25, 6:40 AM

How can i see logs to debug what’s wrong?

HHello, I’m Allie!You can deploy it

Rahul gangotri•2/1/25, 6:42 AM

Tell mw one thing

To reuse sessions and to overcome the 10 session limit

Can i use durable object? Or workflow? Or queue is the best way?

Rahul gangotri•2/1/25, 6:43 AM

Because if all sessions are being used i need to wait and retry

RRahul gangotri How can i see logs to debug what’s wrong?

Hello, I’m Allie!•2/1/25, 8:03 AM

wrangler tailwrangler tail?

Rahul gangotri•2/1/25, 8:05 AM

it does not provide console.log 's

Rahul gangotri•2/1/25, 8:07 AM

even i tried connecting to baselime, it only show basic things like incoming request to queue endpoint and when queue is picked by consumer

that's all

no console logs nothing

o7•2/1/25, 11:03 AM

can Queues be used to batch cache purge a bunch of urls

Oo7 can Queues be used to batch cache purge a bunch of urls

Mitya•2/1/25, 11:51 AM

Sure. The producer worker pushes the URLs into the queue, the consumer worker receives a batch of them, and then calls one of the cache purge API endpoints.

thehungrybird_•2/1/25, 3:16 PM

guys, I'm building a small application with hono.js, but fetch with post method is not working in queues, can anyone help?

RRahul gangotri even i tried connecting to baselime, it only show basic things like incoming req...

aarhus•2/1/25, 9:35 PM

@Rahul gangotri use a service binding to another worker? The other worker has the logging in. I think this works

Aaarhus @Rahul gangotri use a service binding to another worker? The other worker has th...

Rahul gangotri•2/2/25, 5:02 AM

I tried tail worker setup, not printing console logs

hmm, I'm stumped, getting this 403 error message from the api when trying to pull messages from a pa

Similar Threads

Similar Threads

Similar Threads