Yes, Queues does not (by design) deduplicate. That’s often the wrong assumption to make on behalf of

MMatt Silverlock Yes, Queues does not (by design) deduplicate. That’s often the wrong assumption ...

A

Ashley•1/24/24, 1:27 PM

Yeah, I wouldn’t expect it too - would there be a reason for the above, where it built a backlog over 20 mins with 4 messages in the queue, but didn’t consume any? (Based on docs it would have hit the max batch or the timeout way before that)

A

Ashley•1/24/24, 2:19 PM

The metrics in the dashboard are a little slow to update, but it now shows the problem: look at the consumer delay

A

Aster•1/24/24, 3:00 PM

Hey, I am trying to use the graphQL API, but without success

I get an empty response:

{
    "data": {
        "viewer": {
            "accounts": [
                {
                    "queueMessageOperationsAdaptiveGroups": []
                }
            ]
        }
    },
    "errors": null
}

{
    "data": {
        "viewer": {
            "accounts": [
                {
                    "queueMessageOperationsAdaptiveGroups": []
                }
            ]
        }
    },
    "errors": null
}

{
    "data": {
        "viewer": {
            "accounts": [
                {
                    "queueMessageOperationsAdaptiveGroups": []
                }
            ]
        }
    },
    "errors": null
}

{
    "data": {
        "viewer": {
            "accounts": [
                {
                    "queueMessageOperationsAdaptiveGroups": []
                }
            ]
        }
    },
    "errors": null
}

And I used the query

QueueMessageOperationsByMinute

QueueMessageOperationsByMinute

QueueMessageOperationsByMinute

QueueMessageOperationsByMinute from the example (https://developers.cloudflare.com/queues/reference/metrics/), with these variables:

{
    "accountTag": "xxx",
    "queueId": "dev-event-batch-queue",
    "datetimeStart": "2024-01-24T00:00:00Z",
    "datetimeEnd": "2024-01-24T23:59:00Z"
}

{
    "accountTag": "xxx",
    "queueId": "dev-event-batch-queue",
    "datetimeStart": "2024-01-24T00:00:00Z",
    "datetimeEnd": "2024-01-24T23:59:00Z"
}

{
    "accountTag": "xxx",
    "queueId": "dev-event-batch-queue",
    "datetimeStart": "2024-01-24T00:00:00Z",
    "datetimeEnd": "2024-01-24T23:59:00Z"
}

{
    "accountTag": "xxx",
    "queueId": "dev-event-batch-queue",
    "datetimeStart": "2024-01-24T00:00:00Z",
    "datetimeEnd": "2024-01-24T23:59:00Z"
}

Annnd I do get metrics as you can see on the dashboard screenshot (I am sending requests with a curl loop to generate metrics in dev/local env)
Any idea?

AAshley Yeah, I wouldn’t expect it too - would there be a reason for the above, where it...

M

Matt SilverlockOP•1/24/24, 4:29 PM

We think we have a bug here (being a beta) when some messages take a long time to process -> cause delays in subsequent delivery. Working on it.

AAster Hey, I am trying to use the graphQL API, but without success 🤔 I get an empty ...

M

Matt SilverlockOP•1/24/24, 4:30 PM

Have you looked at the queries in https://developers.cloudflare.com/queues/reference/metrics/#get-average-queue-backlog-over-time-period

?

Metrics · Cloudflare Queues

You can view the metrics for a Queue on your account via the Cloudflare dashboard . Navigate to Workers > Queues > your Queue and under the Metrics …

MMatt Silverlock Have you looked at the queries in https://developers.cloudflare.com/queues/refer...

A

Aster•1/24/24, 4:31 PM

Yup that's the link I linked ^^

A

Aster•1/24/24, 4:32 PM

Both 3 queries return an empty array

is maybe the date incorrect? I used 24h tho

AAster Yup that's the link I linked ^^

M

Matt SilverlockOP•1/24/24, 4:33 PM

query {
  viewer {
    accounts(filter: { accountTag: $accountId }) {
      queueMessageOperationsAdaptiveGroups(
        limit: 10000
        filter: { date_geq: $startDate, date_leq: $endDate }
        orderBy: [date_DESC]
      ) {
        count
        sum {
          bytes
        }
        dimensions {
          queueId
          date
        }
      }
    }
  }
}

query {
  viewer {
    accounts(filter: { accountTag: $accountId }) {
      queueMessageOperationsAdaptiveGroups(
        limit: 10000
        filter: { date_geq: $startDate, date_leq: $endDate }
        orderBy: [date_DESC]
      ) {
        count
        sum {
          bytes
        }
        dimensions {
          queueId
          date
        }
      }
    }
  }
}

query {
  viewer {
    accounts(filter: { accountTag: $accountId }) {
      queueMessageOperationsAdaptiveGroups(
        limit: 10000
        filter: { date_geq: $startDate, date_leq: $endDate }
        orderBy: [date_DESC]
      ) {
        count
        sum {
          bytes
        }
        dimensions {
          queueId
          date
        }
      }
    }
  }
}

query {
  viewer {
    accounts(filter: { accountTag: $accountId }) {
      queueMessageOperationsAdaptiveGroups(
        limit: 10000
        filter: { date_geq: $startDate, date_leq: $endDate }
        orderBy: [date_DESC]
      ) {
        count
        sum {
          bytes
        }
        dimensions {
          queueId
          date
        }
      }
    }
  }
}

M

Matt SilverlockOP•1/24/24, 4:33 PM

{
  "data": {
    "viewer": {
      "accounts": [
        {
          "queueMessageOperationsAdaptiveGroups": [
            {
              "count": 2446,
              "dimensions": {
                "date": "2024-01-24",
                "queueId": "e4d7d7e594bd48bbbe49703f72e9c92a"
              },
              "sum": {
                "bytes": 264168
              }
            }
          ]
        }
      ]
    }
  },
  "errors": null
}

{
  "data": {
    "viewer": {
      "accounts": [
        {
          "queueMessageOperationsAdaptiveGroups": [
            {
              "count": 2446,
              "dimensions": {
                "date": "2024-01-24",
                "queueId": "e4d7d7e594bd48bbbe49703f72e9c92a"
              },
              "sum": {
                "bytes": 264168
              }
            }
          ]
        }
      ]
    }
  },
  "errors": null
}

{
  "data": {
    "viewer": {
      "accounts": [
        {
          "queueMessageOperationsAdaptiveGroups": [
            {
              "count": 2446,
              "dimensions": {
                "date": "2024-01-24",
                "queueId": "e4d7d7e594bd48bbbe49703f72e9c92a"
              },
              "sum": {
                "bytes": 264168
              }
            }
          ]
        }
      ]
    }
  },
  "errors": null
}

{
  "data": {
    "viewer": {
      "accounts": [
        {
          "queueMessageOperationsAdaptiveGroups": [
            {
              "count": 2446,
              "dimensions": {
                "date": "2024-01-24",
                "queueId": "e4d7d7e594bd48bbbe49703f72e9c92a"
              },
              "sum": {
                "bytes": 264168
              }
            }
          ]
        }
      ]
    }
  },
  "errors": null
}

M

Matt SilverlockOP•1/24/24, 4:33 PM

(As an example)

M

Matt SilverlockOP•1/24/24, 4:34 PM

I just wrote some messages into my test queue to generate some stats

MMatt Silverlock ``` query { viewer { accounts(filter: { accountTag: $accountId }) { ...

A

Aster•1/24/24, 4:37 PM

Can you share your filters?

A

Aster•1/24/24, 4:40 PM

Note: when I remove the filter by queue, it works

And the date was in wrong format

A

Aster•1/24/24, 4:40 PM

I was using the queue name, but seems like the queue ID is different from the slug

A

Aster•1/24/24, 4:42 PM

How do you get a queue ID ? I dont see any ID from the dashboard (and is it possible to return a slug by any change in the response dimensions?

)

MMatt Silverlock We think we have a bug here (being a beta) when some messages take a long time t...

A

Ashley•1/24/24, 4:45 PM

Thank you, and no worries, not a big deal

Well, in this case it meant I exceeded browser rendering limits

But it's a beta so no sweat, thanks for looking into it!

A

Ashley•1/24/24, 4:46 PM

And what you said makes sense as processing takes ~7 seconds mostly due to time to start Pupeeter + fetch a page

AAster Can you share your filters? 🙂

M

Matt SilverlockOP•1/24/24, 5:00 PM

I don't have any filters: the only filter is

accountTag

accountTag

accountTag

accountTag as defined there.

AAster How do you get a queue ID ? I dont see any ID from the dashboard (and is it poss...

M

Matt SilverlockOP•1/24/24, 5:02 PM

Queue ID:

wrangler queues list

wrangler queues list

wrangler queues list

wrangler queues list today (we need to add it to the dash)

MMatt Silverlock I don't have any filters: the only filter is `accountTag` as defined there.

A

Aster•1/24/24, 5:02 PM

what about

startDate

startDate

startDate

startDate?

MMatt Silverlock Queue ID: `wrangler queues list` today (we need to add it to the dash)

A

Aster•1/24/24, 5:03 PM

Would it be possible to include it in the response dimensions of the graphQL?

AAster what about `startDate`?

M

Matt SilverlockOP•1/24/24, 5:03 PM

A

Aster•1/24/24, 5:04 PM

Sorry variables* my bad

A

Aster•1/24/24, 5:04 PM

Thanks a lot! My main issue is I thought queueId was the queue name ^^

A

Aster•1/25/24, 10:31 AM

Thanks @Matt , it solved my issue btw

A

Aster•1/25/24, 10:31 AM

And for now I'll keep a Map in my code with queue Id and queue name

MMatt Silverlock We think we have a bug here (being a beta) when some messages take a long time t...

A

Ashley•1/25/24, 12:07 PM

Interestingly on this one, the queue shows 1 (out of 4) messages on the dashboard when I poll, but the graph shows 4 in the backlog... I know there's the bug so more just more info. Also, if you manage to fix it, could you let me know it's resolved so I can re-try my implementation please?

A

Ashley•1/25/24, 1:23 PM

Also, related, but... should I be hitting CPU limits when consuming? It seems... really weird I'd be hitting a CPU limit considering I'm on standard pricing, and it should be able to run presumably for... a while? (it is running for at most 10 seconds, but most should be I/O)

A

Adi•1/28/24, 4:23 AM

Hey all, I had a question around the v8 serialization that Queues does to save objects that can't be json serialized.
I'm guessing under the hood it does something like this

const v8 = require('v8');
// Serialize an object into a buffer using v8
function serializeWithV8(obj) {
    return v8.serialize(obj);
}
// Convert the Buffer to a byte array (Uint8Array)
const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
// Now you can send the byteArray over the network

const v8 = require('v8');
// Serialize an object into a buffer using v8
function serializeWithV8(obj) {
    return v8.serialize(obj);
}
// Convert the Buffer to a byte array (Uint8Array)
const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
// Now you can send the byteArray over the network

const v8 = require('v8');
// Serialize an object into a buffer using v8
function serializeWithV8(obj) {
    return v8.serialize(obj);
}
// Convert the Buffer to a byte array (Uint8Array)
const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
// Now you can send the byteArray over the network

const v8 = require('v8');
// Serialize an object into a buffer using v8
function serializeWithV8(obj) {
    return v8.serialize(obj);
}
// Convert the Buffer to a byte array (Uint8Array)
const byteArray = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
// Now you can send the byteArray over the network

However, I can see that the v8 api is not available in workers under

nodejs_compat

nodejs_compat

nodejs_compat

nodejs_compat.

I was looking to making an API interface around queues for a project. Is there a chance that we can get v8 added to nodejs_compat? Is that something the team is already looking into?

K

kian•1/28/24, 4:28 AM

fwiw you can see how it works at https://github.com/cloudflare/workerd/blob/5d27f8f7f1f9b584f673d2f11c9032f5a776ec55/src/workerd/api/queue.c%2B%2B#L33-L128

K

kian•1/28/24, 4:28 AM

it's not done in JS, so the V8 API isn't needed to be exposed

A

Adi•1/28/24, 4:30 AM

Right, that makes sense. I suppose, this is a feature / compatibility request then

Would be great to have v8 available as part of our workers code.

AAdi Right, that makes sense. I suppose, this is a feature / compatibility request th...

H

Hello, I’m Allie!•1/28/24, 9:49 AM

One argument that could be made against there is the fact that the V8 Serialization API is not stable, so for example if you were to serialize something in the browser, send it to a Worker, then try to deserialize it, there is no guarantee that it would suceed

H

Hello, I’m Allie!•1/28/24, 9:50 AM

You would have to ensure that both the browser and the Worker were running the exact same version of the serialisation system, which unless you control all of the clients that request your Worker, would be basically impossible

HHello, I’m Allie!You would have to ensure that both the browser and the Worker were running the e...

A

Adi•1/28/24, 9:54 AM

Oh really? Ok, that's good to know. I was unaware of that complexity, I thought v8 serialization would work in a stable way across node versions

Alright, I'll strike that off my feature list for now then haha
Thank you, I learned a lot

AAdi Oh really? Ok, that's good to know. I was unaware of that complexity, I thought ...

H

Hello, I’m Allie!•1/28/24, 9:56 AM

A newer version of the serializer should be able to read older versions. The issue is when you have an older version trying to read a newer version

H

Hello, I’m Allie!•1/28/24, 9:57 AM

It would cause even more pain were you to store the serialized objects somewhere, as if you ever have to roll back the serializer version, you now have objects that are unreadable

HHello, I’m Allie!It would cause even more pain were you to store the serialized objects somewhere...

A

Adi•1/28/24, 10:01 AM

Yup, that's definitely a big gotcha. And you are spot on, I was planning to save it to KV for retrieval down the line haha

AAdi Yup, that's definitely a big gotcha. And you are spot on, I was planning to save...

H

Hello, I’m Allie!•1/28/24, 10:04 AM

I would probably recommend defining your own serialization format that you control. It doesn't have to compile down to JSON(you could probably make it work with ArrayBuffers), but just the fact that you know/control the format means you can manage upgrades/downgrades yourself

HHello, I’m Allie!I would probably recommend defining your own serialization format that you contr...

A

Adi•1/28/24, 10:07 AM

makes sense
I could probably work up something. Thanks again

A

Aran•1/29/24, 5:44 PM

Are there any known limits with sending a message to a queue while consuming a message from the same queue? I have a pretty simple case where I have a consumer that posts messages to the same queue it is consuming from and the

env.QUEUE.send(...)

env.QUEUE.send(...)

env.QUEUE.send(...)

env.QUEUE.send(...) is called but the consumer never receives the message. No errors. If there isn't a known limit then I'll pair my case down to the most minimal case, but wanted to check here first.

AAran Are there any known limits with sending a message to a queue while consuming a m...

M

Matt SilverlockOP•1/29/24, 5:59 PM

Not a design limitation. A consumer can write back to the same queue (mindful of message loops / potential cost implications!)

A

Aran•1/29/24, 6:00 PM

All right, thanks. Yes, and very aware of the infinite loop possibility.

A

Al Hill•1/30/24, 5:25 PM

I have a producer worker which processes a big array of data and chunk it into pieces to send it to a queue, which is connected to a consumer. I have dimensioned the batches and the messages for not exceeding the maximum batch size, the maximum message size and the maximum subrequest of the consumer worker.
It seems to work, the produces prepares the data and send it to the queue, the queue receives all the messages... and the consumer only receives the first of them. No error in any part of the flow.

I don't know if I'm misunderstanding the queues but I'm pretty confused right now

AAl Hill I have a producer worker which processes a big array of data and chunk it into p...

M

Matt SilverlockOP•1/30/24, 7:36 PM

Do you have a queue ID and account ID you can share?

MMatt Silverlock Do you have a queue ID and account ID you can share?

A

Al Hill•1/31/24, 9:33 AM

PMed

AAl Hill PMed 😉

M

Matt SilverlockOP•1/31/24, 9:34 AM

They are safe to post publicly, btw.

A

Al Hill•1/31/24, 9:40 AM

Ah, ok, I didn't know, haha

A

Al Hill•1/31/24, 9:41 AM

I have being trying to lighten the work of the consumer and divide it in more messages but the problem is the same, only the first one reaches the consumer

A

Al Hill•1/31/24, 9:42 AM

The consumer delay seems to be very very big, but I don't know if I can do anything about this

Yes, Queues does not (by design) deduplicate. That’s often the wrong assumption to make on behalf of

Similar Threads