I am getting this error too in the GUI. So maybe is there something wrong?

Mmaguayo I am getting this error too in the GUI. So maybe is there something wrong?

Jacob•11/28/22, 2:18 PM

Have you published the Worker that contains the exported

queue

queue

handler? Or has it always failed to publish? If so, the GUI would (correctly) think that you don't have a handler

Jacob•11/28/22, 2:19 PM

To test, remove the Consumer from your wrangler.toml, publish the worker, and then try the GUI

maguayoOP•11/28/22, 2:19 PM

The publish is made. The code is live and the fetch is working

maguayoOP•11/28/22, 2:19 PM

Ok, I will try that

maguayoOP•11/28/22, 2:25 PM

Still not working, even removing the consumers and queues from wrnagler. The dashboard doesn't work, but the code is there to handle the queue

maguayoOP•11/28/22, 2:26 PM

Or maybe I am doing the handle wrong:

export default {
  fetch: app.fetch,
  scheduled: async (event, env, ctx) => {
    ctx.waitUntil(cronTrigger(event, env));
  },
  queue: async (batch: MessageBatch<Error>, env: Environment) => {
    fetch("https://api.pushover.net/1/messages.json", {
      method: "POST",
      headers: {
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: new URLSearchParams({
        token: "xxx",
        user: "xxx",
        title: "Hono",
        message: "Example of cron and queue",
      }),
    });
  },
};

export default {
  fetch: app.fetch,
  scheduled: async (event, env, ctx) => {
    ctx.waitUntil(cronTrigger(event, env));
  },
  queue: async (batch: MessageBatch<Error>, env: Environment) => {
    fetch("https://api.pushover.net/1/messages.json", {
      method: "POST",
      headers: {
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: new URLSearchParams({
        token: "xxx",
        user: "xxx",
        title: "Hono",
        message: "Example of cron and queue",
      }),
    });
  },
};

Jacob•11/28/22, 2:27 PM

that fetch should probably be awaited (that likely isn't causing this issue, just an aside)

RRob I would also suggest that sleeping in your consumer Worker is an anti-pattern.

Kai•11/28/22, 2:29 PM

I would agree, but I'm not sure what the alternative is. Right now I'm thinking to just use a batchSize of 1 so thundering herd would be no problem. But I'm really not sure what to do over waiting when I've hit the rate limit. There seems to be no way of blocking the queue otherwise. (Of course I'd prefer to do something like "redeliver this message & don't deliver anything before x", or even just reading from the queue when an alarm is triggered).

Jacob•11/28/22, 2:32 PM

I implemented a throttled queue to manage rate limits for a project in the past where I had to limit to 3rps to the Notion API: https://github.com/jahands/msc-utils/blob/main/src/notion.ts
Could probably use this when processing batches

Kai•11/28/22, 2:33 PM

So currently you could set some sort of semaphore using a DO.

Can you explain how this would work? I can sync the rate limit via one or multiple DOs but this is very complicated to do (especially since I have multiple "layered" rate limits) & DOs aren't really atomic iirc.

JJacob I implemented a throttled queue to manage rate limits for a project in the past ...

Jacob•11/28/22, 2:35 PM

I could see this working pretty well in a Queue consumer, and it feels like a pretty common pattern that you might need to rate-limit processing of items in a batch
It's used like this:

await queue.add(myAsyncFunction)
await queue.clear() // Execute functions in a rate-limited fassion

await queue.add(myAsyncFunction)
await queue.clear() // Execute functions in a rate-limited fassion

p-throttle and p-queue on npm are really helpful packages for stuff like this

RRob At GA we envision something *like* (not the actual API surface - still under con...

Kai•11/28/22, 4:08 PM

Hmm, that would be nice. But wouldn't this be more expensive to the user then just handling the timeout themselves? From what I understand each retry incurs one operation, but consumers are executed as unbound workers, so sleeping for 20s doesn't incurr any charges.

SSananGuliyev I am on enterprise plan. But unfortunately I have no contact to account team.

Unsmart•11/28/22, 6:04 PM

Account team meaning your CSM. Are you the owner of the account? If not I would reach out to whoever at your company is to find out who the CSM is to contact. Or possibly create a support ticket to see if they can help you get access?

Kai•11/28/22, 7:00 PM

Yes, but I'd use retries to bounce off above-ratelimit requests in this scenario, right? It's not that unlikely to have a lot of requests above that rate limit especially during peak hours

Kai•11/28/22, 7:18 PM

that's actually very fair. Specifying larger durations may be viable. Not sure what batch size would do here though? Is there something I'm missing about how batches work?

dave•11/29/22, 11:09 PM

I keep getting removed.

dave•11/29/22, 11:09 PM

I haven’t even logged into my account today.

Ddave I haven’t even logged into my account today.

James•11/29/22, 11:59 PM

I think there are some billing changes happening with how Queues is provisioned on the backend. Hopefully @robsutter can provide more info - sorry if the emails are scary, I know it can be quite worrying when you receive an email about this when you might be using the product

JJames I think there are some billing changes happening with how Queues is provisioned ...

dave•11/30/22, 12:44 AM

oh sweet, yeah, I’m currently using Queues for feeding analytics. Not mission critical yet thankfully.

Erisa•11/30/22, 1:19 AM

Do you still have access to queues or did it remove it properly?

Erisa•11/30/22, 1:20 AM

Shouldn't have actually impacted anything, just a side effect of the billing changes

Kai•11/30/22, 5:38 PM

from workers-and-pages-discussions
Where are consumers placed? I'd be interested in getting a consumer be placed close to some specific region.

Unsmart•11/30/22, 6:49 PM

I could see setting region being useful to set it close to your database so it returns results faster

Unsmart•11/30/22, 6:50 PM

or close to whatever region a specific service/api you use is hosted in if they dont use edge services

Ddave I keep getting removed. 😐

Matt Silverlock•11/30/22, 7:15 PM

Sorry about this: this is a cleanup for some billing stuff. As long as you are on: a) a paid Workers plan and b) enable Queues, it should work. We don't immediately delete your queues.

RRob Long-term we may configure; would love to hear the use case/requirement. CC @Mat...

Kai•11/30/22, 8:02 PM

It's really just an optimization for queues/cron/pubsub, not really a requirement for anything I'm doing. Hinting that there will be outbound requests & to what region would be nice to have.
For my use case, I have a small product that exposes an API and pulls data to serve from external sites. It both does this on demand and on a schedule, to keep upstream data fresh.
It has some complicated internal routing to do based on the upstream resource requested, and one of those routings is what region data is requested from, ie European data is only available from Europe each routing step is another queue and this makes a small tree of queues. To improve latency it'd be great to have Europe specific queues to be attached to European data centers.
I can give more details once I've started actually porting part of the product to cloudflare queues. The current code base is very stateful so specifics don't really translate.

maguayoOP•12/1/22, 9:59 AM

Is it there a way to see how many messages are in a queue? I found nothing in the UI or Cloudflare API

export default { fetch: app.fetch, scheduled: async (event, env, ctx) => { ctx.waitUntil(cronTrigger(event, env)); }, queue: async (batch: MessageBatch<Error>, env: Environment) => { fetch("https://api.pushover.net/1/messages.json", { method: "POST", headers: { "Content-Type": "application/x-www-form-urlencoded", }, body: new URLSearchParams({ token: "xxx", user: "xxx", title: "Hono", message: "Example of cron and queue", }), }); }, };

I am getting this error too in the GUI. So maybe is there something wrong?

Similar Threads

Similar Threads

Similar Threads