I have a producer worker which processes a big array of data and chunk it into pieces to send it to

I have a producer worker which processes a big array of data and chunk it into pieces to send it to a queue, which is connected to a consumer. I have dimensioned the batches and the messages for not exceeding the maximum batch size, the maximum message size and the maximum subrequest of the consumer worker.
It seems to work, the produces prepares the data and send it to the queue, the queue receives all the messages... and the consumer only receives the first of them. No error in any part of the flow.

I don't know if I'm misunderstanding the queues but I'm pretty confused right now

AAl Hill I have a producer worker which processes a big array of data and chunk it into p...

Matt Silverlock•1/30/24, 7:36 PM

Do you have a queue ID and account ID you can share?

MMatt Silverlock Do you have a queue ID and account ID you can share?

Al HillOP•1/31/24, 9:33 AM

PMed

AAl Hill PMed 😉

Matt Silverlock•1/31/24, 9:34 AM

They are safe to post publicly, btw.

Al HillOP•1/31/24, 9:40 AM

Ah, ok, I didn't know, haha

Al HillOP•1/31/24, 9:41 AM

I have being trying to lighten the work of the consumer and divide it in more messages but the problem is the same, only the first one reaches the consumer

Al HillOP•1/31/24, 9:42 AM

The consumer delay seems to be very very big, but I don't know if I can do anything about this

Al HillOP•1/31/24, 9:44 AM

I configured a dead letter queue (named... dead-letter-queue) to check if something went this way and nope, it doesn't receive anything

Yigal•1/31/24, 12:42 PM

Is there any issue running vanilla js with queues? I don't see any examples of it in the docs, only ts

YYigal Is there any issue running vanilla js with queues? I don't see any examples of i...

Hello, I’m Allie!•1/31/24, 12:42 PM

No, there is not

Yigal•1/31/24, 12:47 PM

So, if I have this line:

await kv_storage.put(id, JSON.stringify(data))

await kv_storage.put(id, JSON.stringify(data))

and I want it to go to a queue instead, how would I write that?

kian•1/31/24, 12:50 PM

await env.QUEUE.send(data);

await env.QUEUE.send(data);

kian•1/31/24, 12:50 PM

https://developers.cloudflare.com/queues/get-started/

Yigal•1/31/24, 1:26 PM

I got stuck at figuring out how to declare this interface in js:

export interface Env {
   <MY_QUEUE>: Queue<any>;
}

export interface Env {
   <MY_QUEUE>: Queue<any>;
}

kian•1/31/24, 1:27 PM

Interfaces are just types.

kian•1/31/24, 1:27 PM

You don't need them in JS.

Yigal•1/31/24, 1:41 PM

working, thanks

Al HillOP•1/31/24, 3:36 PM

How can I reduce the consumer delay in a queue?

AAl Hill How can I reduce the consumer delay in a queue?

Matt Silverlock•1/31/24, 4:06 PM

https://developers.cloudflare.com/queues/reference/batching-retries/#batch-settings

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

Yigal•1/31/24, 5:13 PM

I moved a function that edits KV entries inside a queue, but I keep loosing messages, sending a bunch of edits will surely drop some of them.
All of the requests are well received and logged, but the end result is still missing data inside the KV storage.
It's possible that an edit to the same KV will be done more than once per batch, is that an issue?

YYigal I moved a function that edits KV entries inside a queue, but I keep loosing mess...

Yigal•1/31/24, 7:46 PM

Pretty sure caching was the issue

Manav•1/31/24, 8:03 PM

Hi there, im very new to queues! I have been experiencing consumer delays of about 2000+ seconds. I also see in my workers metric that i get "Excedded CPU limits", what i am trying to do in this queue consumer is to parse out website data such as links, and recursively find upto 300 links and then take the data from all the links and ingest it into a vector database.

Tried multiple things
- changed batch size to 1
- changed batch timeout to 1
- max retries to 1

but nothing has helped so far. Any help would be much appreciated

Manav•1/31/24, 8:08 PM

here's my dashboard metrics

Screenshot_2024-01-31_at_12.08.46_PM.png

Aster•2/1/24, 8:44 AM

Hello

Are there more infos about this documentation?
https://developers.cloudflare.com/queues/reference/metrics/#get-message-operations-by-minute
I want to get the latest data (< 1 min) at query time, but I most of the time get an empty response, and need to query 2, 3 min in the past to get results... :/

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

Aster•2/1/24, 8:45 AM

I tried to remove the graphQL dimension "datetimeMinute" so it just groups all recent results

AKStriker98•2/1/24, 6:54 PM

When the public release is ready, is there an expectation on an increase to the throughput per second over the 400 per queue that’s been the beta?

AAKStriker98 When the public release is ready, is there an expectation on an increase to the ...

Matt Silverlock•2/1/24, 7:35 PM

See the pinned messages! queues

AKStriker98•2/1/24, 7:39 PM

Awesome, thanks!

MManav Hi there, im very new to queues! I have been experiencing consumer delays of abo...

Manav•2/1/24, 7:59 PM

hey @Matt are you able to help with this issue? sadly still experiencing it

goldenhelix•2/1/24, 10:56 PM

pages queue consumer

alex35mil•2/3/24, 3:49 PM

Can somebody suggest how to debug on a local machine the following issue? There is a queue that accepts an image url.

max_batch_size

max_batch_size

is set to

.

A consumer:
- downloads an image
- reads its headers to get the dimensions
- puts the image into r2 bucket via binding
- reports the status to a durable object via binding.

When I throw at this queue around 65 images, after ~45 images, the queue gets silently terminated. No errors, no warnings, nothing. It just silently stops processing a current image, and everything stops.

mackenzie•2/5/24, 2:51 PM

I was wondering why the naming of queues is so restrictive compared to KV namespaces etc

Invalid queue name: invalid queue name: STATE-CHANGES. Must match
  ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ [code: 11003]

Invalid queue name: invalid queue name: STATE-CHANGES. Must match
  ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ [code: 11003]

Limited to lowercase and "-" basically.

Mmackenzie I was wondering why the naming of queues is so restrictive compared to KV namesp...

Matt Silverlock•2/5/24, 4:14 PM

Queue names should be valid DNS labels, which this should enforce.

MMatt Silverlock Some updates for @Queues users: - We're working on landing HTTP Pull, which wil...

kcbrewron•2/5/24, 5:51 PM

On the HTTP pull , are there any updates on landing this in the beta

Kkcbrewron On the HTTP pull , are there any updates on landing this in the beta

Matt Silverlock•2/5/24, 5:52 PM

Plan is this month

Alexey•2/7/24, 5:57 PM

Aren't Queue calls Isolated?
I have a simple

queue

queue

function which starts by logging the number of messages logged so far. The

Logger

Logger

is a simple ES6 module I wrote by myself.
If

queue

queue

calls were isolated (by creating a V8 isolate for each call), then the logged number of messages should always be Zero.
But it's not. It's always increasing which means all

queue

queue

calls are being made within the same running environment.
IS it how it's designed?
I was under impression all Workers calls create a new V8 isolate and there is no shared data/memory between the calls.

AAlexey **Aren't Queue calls Isolated? ** I have a simple `queue` function which starts ...

Hello, I’m Allie!•2/7/24, 5:59 PM

It's definitely not guaranteed to be a new isolate

Hello, I’m Allie!•2/7/24, 5:59 PM

If a Worker is set to be called in the same server/datacenter, then there is a reasonable chance that it spawns in the same isolate

HHello, I’m Allie!If a Worker is set to be called in the same server/datacenter, then there is a r...

Alexey•2/7/24, 6:02 PM

Is it the same for all other Worker invocations (fetch, email, cron)?
Looks like I fundamentally misunderstood Workers. I thought every invocation is completely isolated

AAlexey Is it the same for all other Worker invocations (fetch, email, cron)? Looks like...

Hello, I’m Allie!•2/7/24, 6:02 PM

Yes, though for fetches, for example, the likelihood of hitting the same server/datacenter is decently low, so unless you are running a very large-scale application, isolates are only shared between requests from a single client

HHello, I’m Allie!Yes, though for fetches, for example, the likelihood of hitting the same server/...

Alexey•2/7/24, 6:05 PM

hmm, probably here is the point of my confusion. Doesn't it say that isolates are created for each Worker function call?

AAlexey hmm, probably here is the point of my confusion. Doesn't it say that isolates ar...

Hello, I’m Allie!•2/7/24, 6:07 PM

Hm... Would you mind filing an issue for that on https://github.com/cloudflare/cloudflare-docs?

GitHub

GitHub - cloudflare/cloudflare-docs: Cloudflare’s documentation

Cloudflare’s documentation. Contribute to cloudflare/cloudflare-docs development by creating an account on GitHub.

Hello, I’m Allie!•2/7/24, 6:07 PM

And if you have seen that soemwhere else

HHello, I’m Allie!And if you have seen that soemwhere else

Alexey•2/7/24, 6:20 PM

Ok, I'll do. I created a PR there in December and it haven't got any attention yet. So not sure if creating an issue helps

Edit: created the issue https://github.com/cloudflare/cloudflare-docs/issues/12885

GitHub

Wrong statement about V8 isolates created for each Worker function ...

Which Cloudflare product does this pertain to? Workers Existing documentation URL(s) https://developers.cloudflare.com/workers/reference/how-workers-works/ What changes are you suggesting? See scre...

ian•2/8/24, 6:09 PM

Does CF queues have any concept of fanout jobs? The scenario we're trying to support is sorta similar to batching, a parent job for "srape xyz.com" would be queued up, and that job would enqueue a job for each page on the website, and after all children jobs are finished, the parent would be notified in some way.

Is there any way to accomplish this notifiying the parent job?

Here's how Inngest handles the same scenario: https://www.inngest.com/docs/guides/fan-out-jobs

Fan-out (one-to-many) - Inngest Documentation

How to use the fan-out pattern with Inngest to trigger multiple functions from a single event.