Hey, is there no way to work with queues locally?

ShivekOP•7/2/24, 11:06 AM

I'm building an app using KV and Queues. I have set remote=true so I can read env variables from dev workers env, and use dev kv bucket.

ShivekOP•7/2/24, 11:07 AM

But when I introduced Queues, I got this warning and my app breaks down:

▲ [WARNING] Queues are currently in Beta and are not supported in wrangler dev remote mode.

▲ [WARNING] Queues are currently in Beta and are not supported in wrangler dev remote mode.

SShivek Hey, is there no way to work with queues locally?

Hard@Work•7/2/24, 11:24 AM

Sort of. As you noted, you can use them locally, but then you can't connect to any remote resources

HHard@Work Sort of. As you noted, you can use them locally, but then you can't connect to a...

ShivekOP•7/2/24, 11:36 AM

So I turned off remote=true and injected my env variables as .dev.vars. But even then I'm not able to use queues.

ShivekOP•7/2/24, 11:37 AM

  try {
    console.log(1);
    const q = queue.factory(c);
    console.log(1.2);
    await q.send(item);
    console.log(2);
  } catch (error) {
    console.log(3);
    console.log(error);
  }

  try {
    console.log(1);
    const q = queue.factory(c);
    console.log(1.2);
    await q.send(item);
    console.log(2);
  } catch (error) {
    console.log(3);
    console.log(error);
  }

Some sample code. q.sendq.send fails silently. 2 is never printed and I get no errors.

SShivek ```ts try { console.log(1); const q = queue.factory(c); console.lo...

Hard@Work•7/2/24, 11:37 AM

Both the Producer and Consumer must be the same Worker

Hard@Work•7/2/24, 11:37 AM

At least for now

Hard@Work•7/2/24, 11:38 AM

(iirc)

HHard@Work Both the Producer and Consumer must be the same Worker

ShivekOP•7/2/24, 11:39 AM

I have no consumer setup right now. I'm just trying to see if send is working

ShivekOP•7/2/24, 11:39 AM

it is not at the moment

SShivek I have no consumer setup right now. I'm just trying to see if send is working

Hard@Work•7/2/24, 11:39 AM

Yeah, it might fail if there is nothing to write to, I'm not sure

HHard@Work Yeah, it *might* fail if there is nothing to write to, I'm not sure

ShivekOP•7/2/24, 11:45 AM

My config seems correct and I see the resource being attached when I run wrangler dev

ShivekOP•7/2/24, 11:45 AM

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

SShivek ```Your worker has access to the following bindings: - KV Namespaces: - xxxx: ...

Hard@Work•7/2/24, 11:45 AM

Yes, but if there isn't a consumer for that queue, I would guess it might still fail

ShivekOP•7/2/24, 11:46 AM

okay, so both producer and consumers need to be defined before I can write?

SShivek okay, so both producer and consumers need to be defined before I can write?

Hard@Work•7/2/24, 11:47 AM

Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently failing. If you are able to wait a bit, I may be able to test it out and see if there is something else going wrong

HHard@Work Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently faili...

ShivekOP•7/2/24, 11:48 AM

I can wait, I'll try to set a consumer in the meanwhile

ShivekOP•7/2/24, 12:00 PM

@Hard@Work | R2 I'm able to send messages if a consumer is registered. I was assuming that if there are no consumers, the messages will just stay in the Queue. That's not the case. If there are no consumers, the Q didn't register any messages.

PPranshu Maheshwari Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVObo...

Michael Wittig•7/3/24, 8:40 AM

Hi! I received no response so far after filling out the form. How long does it usually take?

johtso•7/3/24, 2:55 PM

Are retries supposed to work when testing locally?

Jjohtso Are retries supposed to work when testing locally?

Pranshu Maheshwari•7/3/24, 9:25 PM

yes! could you share your code if it's not working on your side?

PPranshu Maheshwari yes! could you share your code if it's not working on your side?

johtso•7/3/24, 9:29 PM

ok good to know, let me try and make a super minimal example and double check

johtso•7/3/24, 9:32 PM

ok, that seems to be working!

johtso•7/3/24, 9:35 PM

probably what I was noticing was messages not surviving between dev server reloads

letincho5•7/4/24, 10:49 AM

Hi... I need a recommendation. To improve or maximize performance (message processing per second), which is better: 1 worker with 10 queues (and consumer X in each queue) or 10 workers with 1 queue each, with consumer X in each queue?

mlshvdv•7/4/24, 11:35 AM

@Pranshu Maheshwari Hi! Some questions about Queues:

1. Do you plan to support message deduplication (refusing to publish the same message while it exists in the queue + support for 'force' mode to rewrite it)?
2. Do you have any plans to add an API method to be able to remove a message from the queue by its ID or impedance key?
3. Are you planning to change the maximum delay to 1 year (this is supported by Upstash QStash - very convenient)?
4. Is there an estimated release date for Queues in production?

Thanks!

control_crusher•7/5/24, 3:46 PM

Does anyone have example code of how they handled multiple queues, single worker?

Sounds like, bind multiple in repeat bind blocks. Then switch inside the Queue handler based on which queue the messages came from.

Follow-up question... do batches then include messages from any of the bound queues? If so, do we only define the batch settings in a single bind block? @Pranshu Maheshwari

Ccontrol_crusher Does anyone have example code of how they handled multiple queues, single worker...

Matt Silverlock•7/6/24, 1:04 PM

Producers are entirely separate from consumers.

- You can write to any queues you bind to as a producer.
- Your consumer will receive messages from only the queues it is configured as a consumer for.
- messages will be batched by queue, but each batch can be from any of the queues you have bound to.

Matt Silverlock•7/6/24, 1:05 PM

https://developers.cloudflare.com/queues/configuration/javascript-apis/#consumer and https://developers.cloudflare.com/queues/reference/how-queues-works/#create-a-consumer-worker

Ben•7/8/24, 5:01 PM

When queues go out of beta will they be available under workers free plan?

BBen When queues go out of beta will they be available under workers free plan?

Pranshu Maheshwari•7/8/24, 6:54 PM

we intend to make queues available under the free plan (with limits of course) in the future, although we don't have a timeline yet

PPranshu Maheshwari we intend to make queues available under the free plan (with limits of course) i...

Ben•7/8/24, 6:54 PM

sounds great!

BumblebeeSquare•7/8/24, 8:22 PM

is there message group like feature in queue ? i.e sqs message group

Brett Willis•7/9/24, 4:25 AM

It would be great to be able to manually pause/resume queues (collect from producers but pause dispatch to consumers) and also to purge queue messages. This help both in development and in production e.g. maintenance/migration, disaster recovery, or when dependent services are experiencing downtime. Queues are good for decoupling services for this very reason, and it's good to be able to intervene in emergencies!

minty•7/9/24, 8:30 PM

is it possible to consume messages from a queue from a non-worker piece of code? i.e. the way you can consume messages from SQS using their SDK in the language of your choice via polling/fetching?

Mminty is it possible to consume messages from a queue from a non-worker piece of code?...

Hello, I’m Allie!•7/9/24, 8:37 PM

Yes, see: https://developers.cloudflare.com/queues/configuration/pull-consumers

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

daguang•7/11/24, 10:53 AM

@Hard@Work @HardlyWorkin' Hello, when I use R2 GET data when consumers use R2 GET data, it will cause the queue to complete it directly. I tried to use Try Catch to capture the error, but I found nothing. Is this my configuration file wrong? or a bug ? (And this logic is normal in the normal worker fetch)