Queue ID: `wrangler queues list` today (we need to add it to the dash)

Queue ID:

wrangler queues list

wrangler queues list

today (we need to add it to the dash)

MMatt Silverlock I don't have any filters: the only filter is `accountTag` as defined there.

Aster•1/24/24, 5:02 PM

what about

startDate

startDate

MMatt Silverlock Queue ID: `wrangler queues list` today (we need to add it to the dash)

Aster•1/24/24, 5:03 PM

Would it be possible to include it in the response dimensions of the graphQL?

AAster what about `startDate`?

Matt SilverlockOP•1/24/24, 5:03 PM

Aster•1/24/24, 5:04 PM

Sorry variables* my bad

Aster•1/24/24, 5:04 PM

Thanks a lot! My main issue is I thought queueId was the queue name ^^

Aster•1/25/24, 10:31 AM

Thanks @Matt , it solved my issue btw

Aster•1/25/24, 10:31 AM

And for now I'll keep a Map in my code with queue Id and queue name

MMatt Silverlock We think we have a bug here (being a beta) when some messages take a long time t...

Ashley•1/25/24, 12:07 PM

Interestingly on this one, the queue shows 1 (out of 4) messages on the dashboard when I poll, but the graph shows 4 in the backlog... I know there's the bug so more just more info. Also, if you manage to fix it, could you let me know it's resolved so I can re-try my implementation please?

Ashley•1/25/24, 1:23 PM

Also, related, but... should I be hitting CPU limits when consuming? It seems... really weird I'd be hitting a CPU limit considering I'm on standard pricing, and it should be able to run presumably for... a while? (it is running for at most 10 seconds, but most should be I/O)

Adi•1/28/24, 4:23 AM

Hey all, I had a question around the v8 serialization that Queues does to save objects that can't be json serialized.
I'm guessing under the hood it does something like this

const v8 = require('v8');
// Serialize an object into a buffer using v8
function serializeWithV8(obj) {
    return v8.serialize(obj);
}
// Convert the Buffer to a byte array (Uint8Array)
const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
// Now you can send the byteArray over the network

const v8 = require('v8');
// Serialize an object into a buffer using v8
function serializeWithV8(obj) {
    return v8.serialize(obj);
}
// Convert the Buffer to a byte array (Uint8Array)
const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
// Now you can send the byteArray over the network

However, I can see that the v8 api is not available in workers under

nodejs_compat

nodejs_compat

.

I was looking to making an API interface around queues for a project. Is there a chance that we can get v8 added to nodejs_compat? Is that something the team is already looking into?

kian•1/28/24, 4:28 AM

fwiw you can see how it works at https://github.com/cloudflare/workerd/blob/5d27f8f7f1f9b584f673d2f11c9032f5a776ec55/src/workerd/api/queue.c%2B%2B#L33-L128

kian•1/28/24, 4:28 AM

it's not done in JS, so the V8 API isn't needed to be exposed

Adi•1/28/24, 4:30 AM

Right, that makes sense. I suppose, this is a feature / compatibility request then

Would be great to have v8 available as part of our workers code.

AAdi Right, that makes sense. I suppose, this is a feature / compatibility request th...

Hello, I’m Allie!•1/28/24, 9:49 AM

One argument that could be made against there is the fact that the V8 Serialization API is not stable, so for example if you were to serialize something in the browser, send it to a Worker, then try to deserialize it, there is no guarantee that it would suceed

Hello, I’m Allie!•1/28/24, 9:50 AM

You would have to ensure that both the browser and the Worker were running the exact same version of the serialisation system, which unless you control all of the clients that request your Worker, would be basically impossible

HHello, I’m Allie!You would have to ensure that both the browser and the Worker were running the e...

Adi•1/28/24, 9:54 AM

Oh really? Ok, that's good to know. I was unaware of that complexity, I thought v8 serialization would work in a stable way across node versions

Alright, I'll strike that off my feature list for now then haha
Thank you, I learned a lot

AAdi Oh really? Ok, that's good to know. I was unaware of that complexity, I thought ...

Hello, I’m Allie!•1/28/24, 9:56 AM

A newer version of the serializer should be able to read older versions. The issue is when you have an older version trying to read a newer version

Hello, I’m Allie!•1/28/24, 9:57 AM

It would cause even more pain were you to store the serialized objects somewhere, as if you ever have to roll back the serializer version, you now have objects that are unreadable

HHello, I’m Allie!It would cause even more pain were you to store the serialized objects somewhere...

Adi•1/28/24, 10:01 AM

Yup, that's definitely a big gotcha. And you are spot on, I was planning to save it to KV for retrieval down the line haha

AAdi Yup, that's definitely a big gotcha. And you are spot on, I was planning to save...

Hello, I’m Allie!•1/28/24, 10:04 AM

I would probably recommend defining your own serialization format that you control. It doesn't have to compile down to JSON(you could probably make it work with ArrayBuffers), but just the fact that you know/control the format means you can manage upgrades/downgrades yourself

HHello, I’m Allie!I would probably recommend defining your own serialization format that you contr...

Adi•1/28/24, 10:07 AM

makes sense
I could probably work up something. Thanks again

Aran•1/29/24, 5:44 PM

Are there any known limits with sending a message to a queue while consuming a message from the same queue? I have a pretty simple case where I have a consumer that posts messages to the same queue it is consuming from and the

env.QUEUE.send(...)

env.QUEUE.send(...)

is called but the consumer never receives the message. No errors. If there isn't a known limit then I'll pair my case down to the most minimal case, but wanted to check here first.

AAran Are there any known limits with sending a message to a queue while consuming a m...

Matt SilverlockOP•1/29/24, 5:59 PM

Not a design limitation. A consumer can write back to the same queue (mindful of message loops / potential cost implications!)

Aran•1/29/24, 6:00 PM

All right, thanks. Yes, and very aware of the infinite loop possibility.

Al Hill•1/30/24, 5:25 PM

I have a producer worker which processes a big array of data and chunk it into pieces to send it to a queue, which is connected to a consumer. I have dimensioned the batches and the messages for not exceeding the maximum batch size, the maximum message size and the maximum subrequest of the consumer worker.
It seems to work, the produces prepares the data and send it to the queue, the queue receives all the messages... and the consumer only receives the first of them. No error in any part of the flow.

I don't know if I'm misunderstanding the queues but I'm pretty confused right now

AAl Hill I have a producer worker which processes a big array of data and chunk it into p...

Matt SilverlockOP•1/30/24, 7:36 PM

Do you have a queue ID and account ID you can share?

MMatt Silverlock Do you have a queue ID and account ID you can share?

Al Hill•1/31/24, 9:33 AM

PMed

AAl Hill PMed 😉

Matt SilverlockOP•1/31/24, 9:34 AM

They are safe to post publicly, btw.

Al Hill•1/31/24, 9:40 AM

Ah, ok, I didn't know, haha

Al Hill•1/31/24, 9:41 AM

I have being trying to lighten the work of the consumer and divide it in more messages but the problem is the same, only the first one reaches the consumer

Al Hill•1/31/24, 9:42 AM

The consumer delay seems to be very very big, but I don't know if I can do anything about this

Al Hill•1/31/24, 9:44 AM

I configured a dead letter queue (named... dead-letter-queue) to check if something went this way and nope, it doesn't receive anything

Yigal•1/31/24, 12:42 PM

Is there any issue running vanilla js with queues? I don't see any examples of it in the docs, only ts

YYigal Is there any issue running vanilla js with queues? I don't see any examples of i...

Hello, I’m Allie!•1/31/24, 12:42 PM

No, there is not

Yigal•1/31/24, 12:47 PM

So, if I have this line:

await kv_storage.put(id, JSON.stringify(data))

await kv_storage.put(id, JSON.stringify(data))

and I want it to go to a queue instead, how would I write that?

kian•1/31/24, 12:50 PM

await env.QUEUE.send(data);

await env.QUEUE.send(data);

kian•1/31/24, 12:50 PM

https://developers.cloudflare.com/queues/get-started/

Yigal•1/31/24, 1:26 PM

I got stuck at figuring out how to declare this interface in js:

export interface Env {
   <MY_QUEUE>: Queue<any>;
}

export interface Env {
   <MY_QUEUE>: Queue<any>;
}

kian•1/31/24, 1:27 PM

Interfaces are just types.

kian•1/31/24, 1:27 PM

You don't need them in JS.

Yigal•1/31/24, 1:41 PM

working, thanks

Al Hill•1/31/24, 3:36 PM

How can I reduce the consumer delay in a queue?

AAl Hill How can I reduce the consumer delay in a queue?

Matt SilverlockOP•1/31/24, 4:06 PM

https://developers.cloudflare.com/queues/reference/batching-retries/#batch-settings

Batching and Retries · Cloudflare Queues

When configuring a consumer Worker for a queue, you can also define how messages are batched as they are delivered.

Yigal•1/31/24, 5:13 PM

I moved a function that edits KV entries inside a queue, but I keep loosing messages, sending a bunch of edits will surely drop some of them.
All of the requests are well received and logged, but the end result is still missing data inside the KV storage.
It's possible that an edit to the same KV will be done more than once per batch, is that an issue?

YYigal I moved a function that edits KV entries inside a queue, but I keep loosing mess...

Yigal•1/31/24, 7:46 PM

Pretty sure caching was the issue

Manav•1/31/24, 8:03 PM

Hi there, im very new to queues! I have been experiencing consumer delays of about 2000+ seconds. I also see in my workers metric that i get "Excedded CPU limits", what i am trying to do in this queue consumer is to parse out website data such as links, and recursively find upto 300 links and then take the data from all the links and ingest it into a vector database.

Tried multiple things
- changed batch size to 1
- changed batch timeout to 1
- max retries to 1

but nothing has helped so far. Any help would be much appreciated

Manav•1/31/24, 8:08 PM

here's my dashboard metrics

Screenshot_2024-01-31_at_12.08.46_PM.png

Aster•2/1/24, 8:44 AM

Hello

Are there more infos about this documentation?
https://developers.cloudflare.com/queues/reference/metrics/#get-message-operations-by-minute
I want to get the latest data (< 1 min) at query time, but I most of the time get an empty response, and need to query 2, 3 min in the past to get results... :/

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

Aster•2/1/24, 8:45 AM

I tried to remove the graphQL dimension "datetimeMinute" so it just groups all recent results

AKStriker98•2/1/24, 6:54 PM

When the public release is ready, is there an expectation on an increase to the throughput per second over the 400 per queue that’s been the beta?

const v8 = require('v8'); // Serialize an object into a buffer using v8 function serializeWithV8(obj) { return v8.serialize(obj); } // Convert the Buffer to a byte array (Uint8Array) const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength); // Now you can send the byteArray over the network

Queue ID: `wrangler queues list` today (we need to add it to the dash)

Similar Threads

Similar Threads

Similar Threads