Are you processing jobs in the queue() callback sequentially by the way? You could do some de-duplic

Are you processing jobs in the queue() callback sequentially by the way?
You could do some de-duplication per-batch and concurrently process the batch

AlexOP•10/31/24, 2:48 PM

Assuming your third party can handle that load

AAlex Are you processing jobs in the queue() callback sequentially by the way? You cou...

SteffTek•10/31/24, 2:49 PM

yeah im doing preprocessing, like grouping by ids etc, thats not the issue

SteffTek•10/31/24, 2:49 PM

i only send the type of job and the id of the resource via queue

AlexOP•10/31/24, 2:50 PM

With that preprocessing are you processing those concurrentlcy? Like generate a promise foreach job and use Promise.all() to concurrently process them to speed up the batch processing?

SteffTek•10/31/24, 2:50 PM

yes

SteffTek•10/31/24, 2:50 PM

exactly ^^

AlexOP•10/31/24, 2:50 PM

Ah okay, figured that might also be a cool optimization but you already though about that

SteffTek•10/31/24, 2:52 PM

i just hope the third party api speeds up, like 100ms would be nice ^^

SteffTek•10/31/24, 2:52 PM

currently it takes up to 10 seconds for a single transaction receipt

SteffTek•10/31/24, 2:52 PM

imagine standing at the checkout and waiting 10 seconds for your receipt

AlexOP•10/31/24, 2:52 PM

They might need to learn a bit from your and starting using Workers or something because that is insane

SteffTek•10/31/24, 2:53 PM

and i need to optimize sqlite lol

SteffTek•10/31/24, 2:53 PM

but thats a whole other bag

SteffTek•10/31/24, 2:53 PM

to many sql queries

SteffTek•10/31/24, 2:54 PM

thanks for your insights ^^ have to go

AlexOP•10/31/24, 2:55 PM

np good luck!

letincho5•11/2/24, 12:12 AM

Hi, I've post a performance problem that i already have: https://stackoverflow.com/questions/79149715/cloudflare-queues-with-slow-consumer-issues

Stack Overflow

Cloudflare queues with slow consumer issues

I'm having problems with my Cloudflare queue consumers. Despite using the different configuration options: (max_batch_timeout=0, max_batch_size=1, delaySeconds=0), my workers take between 5 and 10

DanGamble•11/4/24, 9:49 AM

Is there a way to consume/product queues locally when using Vite as well? I've tried having

vite

vite

and

wrangler dev

wrangler dev

running alongside each other but that doesn't seem to work

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

Hello, I’m Allie!•11/4/24, 3:05 PM

You should be able to push into a Queue, assuming your framework supports it

Hello, I’m Allie!•11/4/24, 3:05 PM

But not consume

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

James•11/4/24, 5:16 PM

when running locally i proxy the push to the queue worker's fetch which pushes to its local queue to then also consume

JJames when running locally i proxy the push to the queue worker's fetch which pushes t...

DanGamble•11/4/24, 7:45 PM

Do you have an example of this?

HHello, I’m Allie!You should be able to push into a Queue, assuming your framework supports it

DanGamble•11/4/24, 7:46 PM

Yea, I can push to it but I'm looking to consume it as well. At the moment I'm setting up a 2nd worker and using a service binding which just feels cumbersome

DDanGamble Do you have an example of this?

James•11/4/24, 7:48 PM

my main app is a pages project, and i have a separate worker for consuming the queue, on the qworker, i have a fetch which accepts the same payload as the queue send. if the origin worker is the same as the consumer worker, the queue works as expected locally, i only send to the fetch if my env is local, otherwise my pages app sends directly to the queue

DDanGamble Yea, I can push to it but I'm looking to consume it as well. At the moment I'm s...

Hello, I’m Allie!•11/4/24, 7:49 PM

Wait, so are you using a framework in Vite?

Hello, I’m Allie!•11/4/24, 7:49 PM

Or...

HHello, I’m Allie!Wait, so are you using a framework in Vite?

DanGamble•11/4/24, 7:50 PM

Yea, remix

DDanGamble Yea, remix

Hello, I’m Allie!•11/4/24, 7:50 PM

That is not currently possible, for dev anyway

Hello, I’m Allie!•11/4/24, 7:51 PM

The Frameworks team is working on a way to allow Vite-based dev servers to run in

workerd

workerd

, which would also allow them to expose non-HTTP event handlers(like Queues)

DanGamble•11/4/24, 7:51 PM

I thought as much. I imagine that's this: https://github.com/flarelabs-net/vite-plugin-cloudflare. If so, can't wait for it!

DoggeSlapper•11/7/24, 4:26 PM

Queue wrangler dev concurrency is 1 right?

Baltazar•11/7/24, 5:02 PM

does someone have a code example with pull consumers?

oof2win2•11/7/24, 7:21 PM

is there a way with which i could limit the processing rate of a queue besides with concurrency + batch size? like i want to limit a certain number of messages / h

oof2win2•11/7/24, 7:22 PM

actually nvm i can do this with D1 and a queue table (even though kinda dumb, it works tho)

Ooof2win2 is there a way with which i could limit the processing rate of a queue besides w...

DoggeSlapper•11/7/24, 7:33 PM

u can using pull style consumer

oof2win2•11/7/24, 7:33 PM

i'm using workers so that doesn't really make sense here

oof2win2•11/7/24, 7:34 PM

easier to build my "own" queue via d1 + cron

DDoggeSlapper Queue wrangler dev concurrency is 1 right?

DoggeSlapper•11/7/24, 7:40 PM

Is there a way to run multiple consumers locally?

DoggeSlapper•11/7/24, 7:47 PM

I mean for concurrency not 2 diffrent consumers on same queue, or there is no way in local to have concurrency?

Sam•11/11/24, 2:58 PM

Is there anyway to increase concurrency beyond 250 workers?

I had 900,000 messages to queue last night and it did work but it took over 10 hours to complete all of the workload.

SSam Is there anyway to increase concurrency beyond 250 workers? I had 900,000 mess...

nclevenger•11/11/24, 3:05 PM

Multiple queues?

Ibrahim John•11/13/24, 8:41 AM

Hello guys, I there a way to know that the queue is complete

IIbrahim John Hello guys, I there a way to know that the queue is complete

Matt Silverlock•11/13/24, 11:43 AM

You can see how many messages are in the queue via the dash or the API: https://developers.cloudflare.com/queues/observability/metrics/

Cloudflare Docs

Metrics | Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard

. Navigate to Workers > Queues > your Queue and under the Metrics tab you’ll be able to view line charts describing the number of messages processed by final outcome, the number of messages in the backlog, and other important indicators.

can•11/20/24, 9:29 AM

Hello y'all, happy coding to everybody :pikahey:

I have a question regarding to the getting messages from queue. I have a producer worker which is a queue manager in general which manages routing for queues. basically, if payload { targetQueue: "test-runner", data: {} }{ targetQueue: "test-runner", data: {} }. Let's say I'm sending data to "test-runner" queue via queue manager. For some reasons, sometimes (more oftenly) my consumer worker doesn't get these messages from the queue, it just stay still. Sometimes I'm seeing Exceeded CPU TimeExceeded CPU Time error and that happen right after it runs for the first time and then I can't get anything back from the consumer worker.

In my queue handler i'm doing 2 db operations and making a request to queue manager with the necessary data to keep process going.

What am I doing wrong here? Would love to get some pointers

Thanks in advance!

Ccan Hello y'all, happy coding to everybody :pikahey: I have a question regarding t...

Pranshu Maheshwari•11/20/24, 3:38 PM

It looks like your consumer Worker is running into CPU time limits (documented here)

Pranshu Maheshwari•11/20/24, 3:40 PM

You'll need to change your consumer Worker to handle the messages in under 30s of CPU time! Otherwise your Worker will time out, and the message will fail.

Pranshu Maheshwari•11/20/24, 3:45 PM

For some reasons, sometimes (more oftenly) my consumer worker doesn't get these messages from the queue, it just stay still.

What's likely happening here is that your consumer Worker is taking a long time to process messages, and is erroring out.

Consumer workers autoscale in response to load, but if your Worker is returning errors (such as Exceeded CPU timeExceeded CPU time), they won't auto-scale (see here for details).

Because your consumer worker is erroring out, your consumer isn't autoscaling: it's likely stuck at a concurrency of 1. And so, it might appear that messages aren't being received, but it's more likely that your (single) consumer is stuck processing the previous message.

Cloudflare Docs

Consumer concurrency | Cloudflare Queues

Consumer concurrency allows a consumer Worker processing messages from a queue to automatically scale out horizontally to keep up with the rate that messages are being written to a queue.

pao ramen•11/20/24, 4:21 PM

What does "average backlog" mean? How to see the current backlog? I have a queue that can't stop increasing the average backlog no matter what I do, and I'm concerned. It's at concurrency 1 for some reason.

johtso•11/21/24, 3:59 PM

Is there a way to get the name of the queue from the Queue binding object?

Are you processing jobs in the queue() callback sequentially by the way? You could do some de-duplic

Similar Threads

Similar Threads

Similar Threads