How long have you been using Queues? Trying to understand if you just set it up and saw this, or if

How long have you been using Queues? Trying to understand if you just set it up and saw this, or if you saw this after using Queues for some time.

drk•11/29/23, 1:18 PM

are there good examples of a pages/workers/queue monorepo? i deploy a remix project to pages currently per cloudflare remix guides and want to leverage queues. so far it seems like a single queue is the only documented setup and given remix+pages constraints that makes sense, but i'm curious about additional workers as consumers and my remix app as a producer primarily.

drk•11/29/23, 1:19 PM

and specifically running multiple workers/consumers locally so i can develop end-to-end, but i think i read that you can only run a single queue consumer at a time locally?

drk•11/29/23, 1:20 PM

also looking for tips on general dev workflow for a setup like this. like if it's best to work on consumers in isolation, deploy those (or preview deploy them for dev purposes) and have my remix app running locally use a dev specific queue that's live in cloudflare.

MMatt Silverlock How long have you been using Queues? Trying to understand if you just set it up ...

Shihmin Lee•11/29/23, 1:53 PM

I am just using Queues for a few days. I suspect this is me hitting some rate limits? I also notice a similar issue for Queue + D1.

Shihmin Lee•11/29/23, 1:59 PM

Although I doubt that I was hitting the rate limits since I am not seeing 429, also I only have 2 workers on 2 queue with 1 consumer each inserted like 1000 records over 5 mins on avg.

SShihmin Lee I am just using Queues for a few days. I suspect this is me hitting some rate li...

Matt SilverlockOP•11/29/23, 2:01 PM

Your issue appears identical to queues

Shihmin Lee•11/29/23, 2:01 PM

Gotcha. I've never seen Subrequest depth limit error message though.

SShihmin Lee Gotcha. I've never seen Subrequest depth limit error message though.

Matt SilverlockOP•11/29/23, 4:23 PM

Per that chat message, that error does not always manifest.

TTyler Benfield Is there any known degradation to queues in the last ~12 hours? We are seeing so...

Matt SilverlockOP•11/30/23, 1:14 PM

Not specific to the last 12 hours, but we have had a couple of reports about consumers not being invoked. We’re looking into the root cause.

TTyler Benfield Thanks @Matt! Let us know if there's any information we can provide. We're going...

Matt SilverlockOP•11/30/23, 1:36 PM

Note that it’s not related to concurrency - that just makes it more obvious.

If you have an account ID and queue ID we can add to our list.

MMatt Silverlock Per that chat message, that error does not always manifest.

Shihmin Lee•11/30/23, 3:33 PM

Hey Matt. So do you know if there's a timeline that this is going to be addressed? I'm interested in moving my infra from S3 to using R2, D1, Queues and KVs and such. But so far it's been harsh to see that I am getting so many 503s for combinations of kv, queue, d1, ... etc in the queue msg as well as all the immediate messages that got consumed.

SShihmin Lee Hey Matt. So do you know if there's a timeline that this is going to be addresse...

Matt SilverlockOP•11/30/23, 3:38 PM

We're working on it, but can't give an accurate timeline as it's non-trivial. Days to weeks, not months. Before Queues hits GA.

Shihmin Lee•11/30/23, 3:46 PM

Cool. That would be so nice.

Dj•11/30/23, 5:06 PM

Queues are awesome btw, I just found out about this product sometime last week. Using it to track some very dumb metrics from a Worker and it was very easy to figure out, both consumers and producers.

Dj•11/30/23, 5:07 PM

kian•11/30/23, 8:21 PM

Serializable object

kian•11/30/23, 8:22 PM

https://developers.cloudflare.com/queues/platform/javascript-apis/#queuescontenttype

TTyler Benfield Is there any known degradation to queues in the last ~12 hours? We are seeing so...

Arthur🍃•12/1/23, 1:49 AM

I have the same problem, my queue just stopped being consumed for no apparent reason

Alexander Blöndal•12/1/23, 2:16 PM

Hi! Is there any way of delaying a message, e.g. I want to this message to be consumed in 2 days?

TTyler Benfield @Matt do you see any potential issues with <:this:1010240499464024116> workaroun...

Matt SilverlockOP•12/1/23, 3:43 PM

I don't love the workaround (you shouldn't have to do this!) but there's no issues with it

AAlexander Blöndal Hi! Is there any way of delaying a message, e.g. I want to this message to be co...

Matt SilverlockOP•12/1/23, 3:44 PM

queues

Fun Miles•12/2/23, 11:21 PM

The docs (https://developers.cloudflare.com/workers/platform/pricing/) say queues worker get 15 minutes CPU times but my queue workers have gotten killed way before that (less than 5 minutes of wall time). There's a similar issue with Cron ones. Is this a known problem?

Pricing · Cloudflare Workers docs

Workers plans and pricing information.

RuthsPlace•12/4/23, 9:22 AM

having the same issue @Fun Miles . Any feedback?

FFun Miles The docs (https://developers.cloudflare.com/workers/platform/pricing/) say queue...

Matt SilverlockOP•12/4/23, 2:26 PM

What usage model is your Worker operating as? Also note, from the docs I linked before:

Matt SilverlockOP•12/4/23, 2:26 PM

I am being killed after 32,000ms. Is this a bug in the new billing?

32000ms = 30s (plus some leeway we provide)

Fun Miles•12/4/23, 2:30 PM

30s indeed but a query should go to 15 minutes, ie 900,000 ms.
My query job does D1 batched read requests with await, so most of that shouldn't count as CPU time and then some processing before some batched D1 writes.

FFun Miles 30s indeed but a query should go to 15 minutes, ie 900,000 ms. My query job does...

Matt SilverlockOP•12/4/23, 3:09 PM

Can you answer my other question? What usage model is your Worker running as? From the error you posted, your Queue consumer (I think?) is hitting the CPU limit posted above:

Queue schedule-retrieval (7 messages) - Exceeded CPU Limit @ 12/2/2023, 12:03:45 PM

Queue schedule-retrieval (7 messages) - Exceeded CPU Limit @ 12/2/2023, 12:03:45 PM

MMatt Silverlock Can you answer my other question? What usage model is your Worker running as? Fr...

Fun Miles•12/4/23, 3:10 PM

I don't understand what you mean by "model" then.

Fun Miles•12/4/23, 3:12 PM

It's the standard billing which says:

Fun Miles•12/4/23, 3:13 PM

As I've hit the limit both in Cron Trigger, then, after refactoring the code to divide in smaller queue works, the Queue Consumer, I am presuming there is a bug and it probably has the same root in both.

FFun Miles I don't understand what you mean by "model" then.

Matt SilverlockOP•12/4/23, 3:15 PM

https://developers.cloudflare.com/workers/platform/pricing/#how-to-switch-usage-model

your Worker will run under a usage model. You can check the dash. It will depend on when you created it.

Fun Miles•12/4/23, 3:17 PM

Standard model.

Fun Miles•12/4/23, 3:40 PM

RuthsPlace•12/4/23, 6:36 PM

For reference, the only way I could get more than 30 seconds was by
1) Moving from "standard" to "unbound"
2) Moving to "cron" trigger (I realize this is likely not helpful if you need queue but could work with my use case)
3) Adding this line to my wrangler.toml - workers_dev = false

RuthsPlace•12/4/23, 6:38 PM

You can see the success logs. The green ones are where the only change is to run under "unbound" usage model. The failed ones are where I run "standard" usage model.

RuthsPlace•12/4/23, 6:40 PM

Anyhow, just wanted to share in case it helps Cloudflare team diagnose issue. I don't think it's behaving as the documentation expresses it currently, unless I'm missing something

Ryan K•12/5/23, 12:05 AM

Hi can anyone help provide more context for this error I'm seeing when deploying workers with a queue:

Total Upload: 1810.83 KiB / gzip: 347.18 KiB
Uploaded dev-api-internal (2.44 sec)

✘ [ERROR] A request to the Cloudflare API (/accounts/2c/workers/queues/dev-queue/consumers/api-backend) failed.

  workers.api.error.queue_handler_missing [code: 11001]

Total Upload: 1810.83 KiB / gzip: 347.18 KiB
Uploaded dev-api-internal (2.44 sec)

✘ [ERROR] A request to the Cloudflare API (/accounts/2c/workers/queues/dev-queue/consumers/api-backend) failed.

  workers.api.error.queue_handler_missing [code: 11001]

kian•12/5/23, 12:10 AM

It's just saying that it doesn't detect a

queue

queue

function in your default export

Kkian It's just saying that it doesn't detect a `queue` function in your default expor...

Ryan K•12/5/23, 12:13 AM

I assumed that could be the case but I do indeed have a default export:

export default {
  fetch(request: Request, env: Env, ctx: ExecutionContext) {
    // Set env variables
    Object.keys(env).forEach((variable) => {
      process.env[variable] ??= `${env[variable]}`;
    })

    return fetch(request, env, ctx)
  },
  async queue(batch: MessageBatch<any>, env: Env): Promise<void> {
    let messages = JSON.stringify(batch.messages);
    console.log(`consumed from our queue: ${messages}`);
  },
}

export default {
  fetch(request: Request, env: Env, ctx: ExecutionContext) {
    // Set env variables
    Object.keys(env).forEach((variable) => {
      process.env[variable] ??= `${env[variable]}`;
    })

    return fetch(request, env, ctx)
  },
  async queue(batch: MessageBatch<any>, env: Env): Promise<void> {
    let messages = JSON.stringify(batch.messages);
    console.log(`consumed from our queue: ${messages}`);
  },
}

Total Upload: 1810.83 KiB / gzip: 347.18 KiB Uploaded dev-api-internal (2.44 sec) ✘ [ERROR] A request to the Cloudflare API (/accounts/2c/workers/queues/dev-queue/consumers/api-backend) failed. workers.api.error.queue_handler_missing [code: 11001]

export default { fetch(request: Request, env: Env, ctx: ExecutionContext) { // Set env variables Object.keys(env).forEach((variable) => { process.env[variable] ??= `${env[variable]}`; }) return fetch(request, env, ctx) }, async queue(batch: MessageBatch<any>, env: Env): Promise<void> { let messages = JSON.stringify(batch.messages); console.log(`consumed from our queue: ${messages}`); }, }

How long have you been using Queues? Trying to understand if you just set it up and saw this, or if

Similar Threads

Similar Threads

Similar Threads