Can you try adding `max_retries` to your consumer? Also try increasing `max_batch_size`? Just to con

Can you try adding

max_retries

max_retries

to your consumer? Also try increasing

max_batch_size

max_batch_size

? Just to confirm there's not a bug with those values

JJacob Can you try adding `max_retries` to your consumer? Also try increasing `max_batc...

maguayo•11/28/22, 2:16 PM

Tried that, and not working either. But what worries me, is that even the GUI:
https://dash.cloudflare.com/xxxxxxxx/workers/services/view/xxxxxx/production/triggers

is not working

maguayo•11/28/22, 2:17 PM

I am getting this error too in the GUI. So maybe is there something wrong?

Mmaguayo I am getting this error too in the GUI. So maybe is there something wrong?

JacobOP•11/28/22, 2:18 PM

Have you published the Worker that contains the exported

queue

queue

handler? Or has it always failed to publish? If so, the GUI would (correctly) think that you don't have a handler

JacobOP•11/28/22, 2:19 PM

To test, remove the Consumer from your wrangler.toml, publish the worker, and then try the GUI

maguayo•11/28/22, 2:19 PM

The publish is made. The code is live and the fetch is working

maguayo•11/28/22, 2:19 PM

Ok, I will try that

maguayo•11/28/22, 2:25 PM

Still not working, even removing the consumers and queues from wrnagler. The dashboard doesn't work, but the code is there to handle the queue

maguayo•11/28/22, 2:26 PM

Or maybe I am doing the handle wrong:

export default {
  fetch: app.fetch,
  scheduled: async (event, env, ctx) => {
    ctx.waitUntil(cronTrigger(event, env));
  },
  queue: async (batch: MessageBatch<Error>, env: Environment) => {
    fetch("https://api.pushover.net/1/messages.json", {
      method: "POST",
      headers: {
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: new URLSearchParams({
        token: "xxx",
        user: "xxx",
        title: "Hono",
        message: "Example of cron and queue",
      }),
    });
  },
};

export default {
  fetch: app.fetch,
  scheduled: async (event, env, ctx) => {
    ctx.waitUntil(cronTrigger(event, env));
  },
  queue: async (batch: MessageBatch<Error>, env: Environment) => {
    fetch("https://api.pushover.net/1/messages.json", {
      method: "POST",
      headers: {
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: new URLSearchParams({
        token: "xxx",
        user: "xxx",
        title: "Hono",
        message: "Example of cron and queue",
      }),
    });
  },
};

JacobOP•11/28/22, 2:27 PM

that fetch should probably be awaited (that likely isn't causing this issue, just an aside)

RRob I would also suggest that sleeping in your consumer Worker is an anti-pattern.

Kai•11/28/22, 2:29 PM

I would agree, but I'm not sure what the alternative is. Right now I'm thinking to just use a batchSize of 1 so thundering herd would be no problem. But I'm really not sure what to do over waiting when I've hit the rate limit. There seems to be no way of blocking the queue otherwise. (Of course I'd prefer to do something like "redeliver this message & don't deliver anything before x", or even just reading from the queue when an alarm is triggered).

JacobOP•11/28/22, 2:32 PM

I implemented a throttled queue to manage rate limits for a project in the past where I had to limit to 3rps to the Notion API: https://github.com/jahands/msc-utils/blob/main/src/notion.ts
Could probably use this when processing batches

Kai•11/28/22, 2:33 PM

So currently you could set some sort of semaphore using a DO.

Can you explain how this would work? I can sync the rate limit via one or multiple DOs but this is very complicated to do (especially since I have multiple "layered" rate limits) & DOs aren't really atomic iirc.

JJacob I implemented a throttled queue to manage rate limits for a project in the past ...

JacobOP•11/28/22, 2:35 PM

I could see this working pretty well in a Queue consumer, and it feels like a pretty common pattern that you might need to rate-limit processing of items in a batch
It's used like this:

await queue.add(myAsyncFunction)
await queue.clear() // Execute functions in a rate-limited fassion

await queue.add(myAsyncFunction)
await queue.clear() // Execute functions in a rate-limited fassion

p-throttle and p-queue on npm are really helpful packages for stuff like this

RRob At GA we envision something *like* (not the actual API surface - still under con...

Kai•11/28/22, 4:08 PM

Hmm, that would be nice. But wouldn't this be more expensive to the user then just handling the timeout themselves? From what I understand each retry incurs one operation, but consumers are executed as unbound workers, so sleeping for 20s doesn't incurr any charges.

SSananGuliyev I am on enterprise plan. But unfortunately I have no contact to account team.

Unsmart•11/28/22, 6:04 PM

Account team meaning your CSM. Are you the owner of the account? If not I would reach out to whoever at your company is to find out who the CSM is to contact. Or possibly create a support ticket to see if they can help you get access?

Kai•11/28/22, 7:00 PM

Yes, but I'd use retries to bounce off above-ratelimit requests in this scenario, right? It's not that unlikely to have a lot of requests above that rate limit especially during peak hours

Kai•11/28/22, 7:18 PM

that's actually very fair. Specifying larger durations may be viable. Not sure what batch size would do here though? Is there something I'm missing about how batches work?

dave•11/29/22, 11:09 PM

I keep getting removed.

dave•11/29/22, 11:09 PM

I haven’t even logged into my account today.

Ddave I haven’t even logged into my account today.

James•11/29/22, 11:59 PM

I think there are some billing changes happening with how Queues is provisioned on the backend. Hopefully @robsutter can provide more info - sorry if the emails are scary, I know it can be quite worrying when you receive an email about this when you might be using the product

JJames I think there are some billing changes happening with how Queues is provisioned ...

dave•11/30/22, 12:44 AM

oh sweet, yeah, I’m currently using Queues for feeding analytics. Not mission critical yet thankfully.

Erisa•11/30/22, 1:19 AM

Do you still have access to queues or did it remove it properly?

Erisa•11/30/22, 1:20 AM

Shouldn't have actually impacted anything, just a side effect of the billing changes

Kai•11/30/22, 5:38 PM

from workers-and-pages-discussions
Where are consumers placed? I'd be interested in getting a consumer be placed close to some specific region.

Unsmart•11/30/22, 6:49 PM

I could see setting region being useful to set it close to your database so it returns results faster

Unsmart•11/30/22, 6:50 PM

or close to whatever region a specific service/api you use is hosted in if they dont use edge services

Ddave I keep getting removed. 😐

Matt Silverlock•11/30/22, 7:15 PM

Sorry about this: this is a cleanup for some billing stuff. As long as you are on: a) a paid Workers plan and b) enable Queues, it should work. We don't immediately delete your queues.

export default { fetch: app.fetch, scheduled: async (event, env, ctx) => { ctx.waitUntil(cronTrigger(event, env)); }, queue: async (batch: MessageBatch<Error>, env: Environment) => { fetch("https://api.pushover.net/1/messages.json", { method: "POST", headers: { "Content-Type": "application/x-www-form-urlencoded", }, body: new URLSearchParams({ token: "xxx", user: "xxx", title: "Hono", message: "Example of cron and queue", }), }); }, };

Can you try adding `max_retries` to your consumer? Also try increasing `max_batch_size`? Just to con

Similar Threads

Similar Threads

Similar Threads