Cloudflare Developers•2y ago

How can I reduce the consumer delay in a queue?

AAl Hill How can I reduce the consumer delay in a queue?

Matt Silverlock•1/31/24, 4:06 PM

https://developers.cloudflare.com/queues/reference/batching-retries/#batch-settings

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

Yigal•1/31/24, 5:13 PM

I moved a function that edits KV entries inside a queue, but I keep loosing messages, sending a bunch of edits will surely drop some of them.
All of the requests are well received and logged, but the end result is still missing data inside the KV storage.
It's possible that an edit to the same KV will be done more than once per batch, is that an issue?

YYigal I moved a function that edits KV entries inside a queue, but I keep loosing mess...

Yigal•1/31/24, 7:46 PM

Pretty sure caching was the issue

Manav•1/31/24, 8:03 PM

Hi there, im very new to queues! I have been experiencing consumer delays of about 2000+ seconds. I also see in my workers metric that i get "Excedded CPU limits", what i am trying to do in this queue consumer is to parse out website data such as links, and recursively find upto 300 links and then take the data from all the links and ingest it into a vector database.

Tried multiple things
- changed batch size to 1
- changed batch timeout to 1
- max retries to 1

but nothing has helped so far. Any help would be much appreciated

Manav•1/31/24, 8:08 PM

here's my dashboard metrics

Screenshot_2024-01-31_at_12.08.46_PM.png

Aster•2/1/24, 8:44 AM

Hello

Are there more infos about this documentation?
https://developers.cloudflare.com/queues/reference/metrics/#get-message-operations-by-minute
I want to get the latest data (< 1 min) at query time, but I most of the time get an empty response, and need to query 2, 3 min in the past to get results... :/

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

Aster•2/1/24, 8:45 AM

I tried to remove the graphQL dimension "datetimeMinute" so it just groups all recent results

AKStriker98•2/1/24, 6:54 PM

When the public release is ready, is there an expectation on an increase to the throughput per second over the 400 per queue that’s been the beta?

AAKStriker98 When the public release is ready, is there an expectation on an increase to the ...

Matt Silverlock•2/1/24, 7:35 PM

See the pinned messages! queues

AKStriker98•2/1/24, 7:39 PM

Awesome, thanks!

MManav Hi there, im very new to queues! I have been experiencing consumer delays of abo...

Manav•2/1/24, 7:59 PM

hey @Matt are you able to help with this issue? sadly still experiencing it

goldenhelix•2/1/24, 10:56 PM

pages queue consumer

alex35mil•2/3/24, 3:49 PM

Can somebody suggest how to debug on a local machine the following issue? There is a queue that accepts an image url. max_batch_sizemax_batch_size is set to 11.

A consumer:
- downloads an image
- reads its headers to get the dimensions
- puts the image into r2 bucket via binding
- reports the status to a durable object via binding.

When I throw at this queue around 65 images, after ~45 images, the queue gets silently terminated. No errors, no warnings, nothing. It just silently stops processing a current image, and everything stops.

mackenzie•2/5/24, 2:51 PM

I was wondering why the naming of queues is so restrictive compared to KV namespaces etc

Invalid queue name: invalid queue name: STATE-CHANGES. Must match
  ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ [code: 11003]

Invalid queue name: invalid queue name: STATE-CHANGES. Must match
  ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ [code: 11003]

Limited to lowercase and "-" basically.

Mmackenzie I was wondering why the naming of queues is so restrictive compared to KV namesp...

Matt Silverlock•2/5/24, 4:14 PM

Queue names should be valid DNS labels, which this should enforce.

MMatt Silverlock Some updates for @Queues users: - We're working on landing HTTP Pull, which wil...

kcbrewron•2/5/24, 5:51 PM

On the HTTP pull , are there any updates on landing this in the beta

Kkcbrewron On the HTTP pull , are there any updates on landing this in the beta

Matt Silverlock•2/5/24, 5:52 PM

Plan is this month

Alexey•2/7/24, 5:57 PM

Aren't Queue calls Isolated?
I have a simple queuequeue function which starts by logging the number of messages logged so far. The LoggerLogger is a simple ES6 module I wrote by myself.
If queuequeue calls were isolated (by creating a V8 isolate for each call), then the logged number of messages should always be Zero.
But it's not. It's always increasing which means all queuequeue calls are being made within the same running environment.
IS it how it's designed?
I was under impression all Workers calls create a new V8 isolate and there is no shared data/memory between the calls.

AAlexey **Aren't Queue calls Isolated? ** I have a simple `queue` function which starts ...

Hello, I’m Allie!•2/7/24, 5:59 PM

It's definitely not guaranteed to be a new isolate

Hello, I’m Allie!•2/7/24, 5:59 PM

If a Worker is set to be called in the same server/datacenter, then there is a reasonable chance that it spawns in the same isolate

HHello, I’m Allie!If a Worker is set to be called in the same server/datacenter, then there is a r...

Alexey•2/7/24, 6:02 PM

Is it the same for all other Worker invocations (fetch, email, cron)?
Looks like I fundamentally misunderstood Workers. I thought every invocation is completely isolated

AAlexey Is it the same for all other Worker invocations (fetch, email, cron)? Looks like...

Hello, I’m Allie!•2/7/24, 6:02 PM

Yes, though for fetches, for example, the likelihood of hitting the same server/datacenter is decently low, so unless you are running a very large-scale application, isolates are only shared between requests from a single client

HHello, I’m Allie!Yes, though for fetches, for example, the likelihood of hitting the same server/...

Alexey•2/7/24, 6:05 PM

hmm, probably here is the point of my confusion. Doesn't it say that isolates are created for each Worker function call?

AAlexey hmm, probably here is the point of my confusion. Doesn't it say that isolates ar...

Hello, I’m Allie!•2/7/24, 6:07 PM

Hm... Would you mind filing an issue for that on https://github.com/cloudflare/cloudflare-docs?

GitHub

GitHub - cloudflare/cloudflare-docs: Cloudflare’s documentation

Cloudflare’s documentation. Contribute to cloudflare/cloudflare-docs development by creating an account on GitHub.

Hello, I’m Allie!•2/7/24, 6:07 PM

And if you have seen that soemwhere else

HHello, I’m Allie!And if you have seen that soemwhere else

Alexey•2/7/24, 6:20 PM

Ok, I'll do. I created a PR there in December and it haven't got any attention yet. So not sure if creating an issue helps

Edit: created the issue https://github.com/cloudflare/cloudflare-docs/issues/12885

GitHub

Wrong statement about V8 isolates created for each Worker function ...

Which Cloudflare product does this pertain to? Workers Existing documentation URL(s) https://developers.cloudflare.com/workers/reference/how-workers-works/ What changes are you suggesting? See scre...

ian•2/8/24, 6:09 PM

Does CF queues have any concept of fanout jobs? The scenario we're trying to support is sorta similar to batching, a parent job for "srape xyz.com" would be queued up, and that job would enqueue a job for each page on the website, and after all children jobs are finished, the parent would be notified in some way.

Is there any way to accomplish this notifiying the parent job?

Here's how Inngest handles the same scenario: https://www.inngest.com/docs/guides/fan-out-jobs

Fan-out (one-to-many) - Inngest Documentation

How to use the fan-out pattern with Inngest to trigger multiple functions from a single event.

Alvee•2/9/24, 12:43 AM

how is queues . anyone using in production?

Jadugar_Jaggu•2/9/24, 4:45 PM

Hey there does consumer queue have 15 minutes of cpu time per invocation?

Jadugar_Jaggu•2/9/24, 4:46 PM

or is there any hidden condition?

Alvee•2/10/24, 11:05 AM

when will queue remove beta and ready for production

AAlvee when will queue remove beta and ready for production

Matt Silverlock•2/10/24, 12:06 PM

queues

Rrenzor Did http pull ever land?

John Spurlock•2/11/24, 3:36 PM

queues

MMatt Silverlock Plan is this month 🙂

switz•2/13/24, 9:38 PM

just wondering, does this also mean that queues can hold items for consumption later? if so, how long will they remain in the queue?

switz•2/13/24, 9:39 PM

not just delay, I mean if the http pull isn't available and doesn't come online for 1h-24h

Sswitz not just delay, I mean if the http pull isn't available and doesn't come online ...

Matt Silverlock•2/13/24, 9:41 PM

If you don’t consume from a Pull consumer they will stay in the queue for the retention limit - 96 hours: https://developers.cloudflare.com/queues/platform/limits/

Cloudflare Docs

Limits · Cloudflare Queues

Many of these limits will increase during Queues’ public beta period. Follow our changelog or join the #queues-beta channel in our Developer Discord …

switz•2/13/24, 9:42 PM

fantastic, thank you for the info

switz•2/13/24, 9:43 PM

just to drop a link as you guys work on http pull, want to share some prior art in sqs-consumer which I find to be a very helpful API https://github.com/bbc/sqs-consumer

GitHub

GitHub - bbc/sqs-consumer: Build Amazon Simple Queue Service (SQS) ...

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate - bbc/sqs-consumer

switz•2/13/24, 9:44 PM

unclear if you'll release a client library alongside http pull or its just an api, but I like using this with sqs

dattaproffs•2/14/24, 8:32 PM

hi, is it bad practice to have a worker with both a

fetch()

fetch()

and a queue()queue() ? the fetch is a simple web page. I'm have a litte troubble understanding what happens when the queuequeue forces the worker to scale? Is there any potential problems there?

Ddattaproffs hi, is it bad practice to have a worker with both a `fetch()` and a `queue()` ? ...

Hello, I’m Allie!•2/14/24, 10:15 PM

Nope, not at all. You can have every handler in a single Worker if you want

HHello, I’m Allie!Nope, not at all. You can have every handler in a single Worker if you want

dattaproffs•2/14/24, 10:17 PM

That's what i thought, but couldn't find any direct confirmation. anywhere in docs or so i can read about it? Thank's for the answer

Yigal•2/15/24, 6:28 AM

I have a situation where I have batches writing to KV, but if there are more than one update to the same key in one batch, the KV is likely to be overwritten and miss updates.
I thought it might be useful to hold the operation of the next batch\message until the first one ends.
How would you go about this kind of issue?

Yigal•2/15/24, 5:04 PM

Is there a way to control the time gap between each batch?

YYigal Is there a way to control the time gap between each batch?

Matt Silverlock•2/15/24, 11:33 PM

You can set the batch interval. If you need to delay messages, then see queues

piffie•2/22/24, 2:31 PM

Are there statistics available how long it takes for a queue to send it to an consumer? Is it typically less than a second? (not counting retries or something, just the typical case)

How can I reduce the consumer delay in a queue?

Similar Threads

Similar Threads

Similar Threads