Cloudflare Developers•2y ago

Both the Producer and Consumer must be the same Worker

Hard@WorkOP•7/2/24, 11:37 AM

At least for now

Hard@WorkOP•7/2/24, 11:38 AM

(iirc)

HHard@Work Both the Producer and Consumer must be the same Worker

Shivek•7/2/24, 11:39 AM

I have no consumer setup right now. I'm just trying to see if send is working

Shivek•7/2/24, 11:39 AM

it is not at the moment

SShivek I have no consumer setup right now. I'm just trying to see if send is working

Hard@WorkOP•7/2/24, 11:39 AM

Yeah, it might fail if there is nothing to write to, I'm not sure

HHard@Work Yeah, it *might* fail if there is nothing to write to, I'm not sure

Shivek•7/2/24, 11:45 AM

My config seems correct and I see the resource being attached when I run wrangler dev

Shivek•7/2/24, 11:45 AM

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

Your worker has access to the following bindings:
- KV Namespaces:
  - xxxx: xxxx
- Queues:
  - TREASURY_QUEUE: dev-treasury-queue

SShivek ```Your worker has access to the following bindings: - KV Namespaces: - xxxx: ...

Hard@WorkOP•7/2/24, 11:45 AM

Yes, but if there isn't a consumer for that queue, I would guess it might still fail

Shivek•7/2/24, 11:46 AM

okay, so both producer and consumers need to be defined before I can write?

SShivek okay, so both producer and consumers need to be defined before I can write?

Hard@WorkOP•7/2/24, 11:47 AM

Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently failing. If you are able to wait a bit, I may be able to test it out and see if there is something else going wrong

HHard@Work Maybe? Tbh, I find it odd that it isn't erroring at all, but just silently faili...

Shivek•7/2/24, 11:48 AM

I can wait, I'll try to set a consumer in the meanwhile

Shivek•7/2/24, 12:00 PM

@Hard@Work | R2 I'm able to send messages if a consumer is registered. I was assuming that if there are no consumers, the messages will just stay in the Queue. That's not the case. If there are no consumers, the Q didn't register any messages.

PPranshu Maheshwari Could you fill out this form?https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVObo...

Michael Wittig•7/3/24, 8:40 AM

Hi! I received no response so far after filling out the form. How long does it usually take?

johtso•7/3/24, 2:55 PM

Are retries supposed to work when testing locally?

Jjohtso Are retries supposed to work when testing locally?

Pranshu Maheshwari•7/3/24, 9:25 PM

yes! could you share your code if it's not working on your side?

PPranshu Maheshwari yes! could you share your code if it's not working on your side?

johtso•7/3/24, 9:29 PM

ok good to know, let me try and make a super minimal example and double check

johtso•7/3/24, 9:32 PM

ok, that seems to be working!

johtso•7/3/24, 9:35 PM

probably what I was noticing was messages not surviving between dev server reloads

letincho5•7/4/24, 10:49 AM

Hi... I need a recommendation. To improve or maximize performance (message processing per second), which is better: 1 worker with 10 queues (and consumer X in each queue) or 10 workers with 1 queue each, with consumer X in each queue?

mlshvdv•7/4/24, 11:35 AM

@Pranshu Maheshwari Hi! Some questions about Queues:

1. Do you plan to support message deduplication (refusing to publish the same message while it exists in the queue + support for 'force' mode to rewrite it)?
2. Do you have any plans to add an API method to be able to remove a message from the queue by its ID or impedance key?
3. Are you planning to change the maximum delay to 1 year (this is supported by Upstash QStash - very convenient)?
4. Is there an estimated release date for Queues in production?

Thanks!

control_crusher•7/5/24, 3:46 PM

Does anyone have example code of how they handled multiple queues, single worker?

Sounds like, bind multiple in repeat bind blocks. Then switch inside the Queue handler based on which queue the messages came from.

Follow-up question... do batches then include messages from any of the bound queues? If so, do we only define the batch settings in a single bind block? @Pranshu Maheshwari

Ccontrol_crusher Does anyone have example code of how they handled multiple queues, single worker...

Matt Silverlock•7/6/24, 1:04 PM

Producers are entirely separate from consumers.

- You can write to any queues you bind to as a producer.
- Your consumer will receive messages from only the queues it is configured as a consumer for.
- messages will be batched by queue, but each batch can be from any of the queues you have bound to.

Matt Silverlock•7/6/24, 1:05 PM

https://developers.cloudflare.com/queues/configuration/javascript-apis/#consumer and https://developers.cloudflare.com/queues/reference/how-queues-works/#create-a-consumer-worker

Ben•7/8/24, 5:01 PM

When queues go out of beta will they be available under workers free plan?

BBen When queues go out of beta will they be available under workers free plan?

Pranshu Maheshwari•7/8/24, 6:54 PM

we intend to make queues available under the free plan (with limits of course) in the future, although we don't have a timeline yet

PPranshu Maheshwari we intend to make queues available under the free plan (with limits of course) i...

Ben•7/8/24, 6:54 PM

sounds great!

BumblebeeSquare•7/8/24, 8:22 PM

is there message group like feature in queue ? i.e sqs message group

Brett Willis•7/9/24, 4:25 AM

It would be great to be able to manually pause/resume queues (collect from producers but pause dispatch to consumers) and also to purge queue messages. This help both in development and in production e.g. maintenance/migration, disaster recovery, or when dependent services are experiencing downtime. Queues are good for decoupling services for this very reason, and it's good to be able to intervene in emergencies!

minty•7/9/24, 8:30 PM

is it possible to consume messages from a queue from a non-worker piece of code? i.e. the way you can consume messages from SQS using their SDK in the language of your choice via polling/fetching?

Mminty is it possible to consume messages from a queue from a non-worker piece of code?...

Hello, I’m Allie!•7/9/24, 8:37 PM

Yes, see: https://developers.cloudflare.com/queues/configuration/pull-consumers

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

daguang•7/11/24, 10:53 AM

@Hard@Work @HardlyWorkin' Hello, when I use R2 GET data when consumers use R2 GET data, it will cause the queue to complete it directly. I tried to use Try Catch to capture the error, but I found nothing. Is this my configuration file wrong? or a bug ? (And this logic is normal in the normal worker fetch)

kian•7/11/24, 4:41 PM

you're returning errors rather than throwing them?

kian•7/11/24, 4:41 PM

:meowthinking:

Ccarlos-alberto Since ~18:00 yesterday one of our queues has been erroring with `Queue send fail...

ajgeiss0702•7/11/24, 11:38 PM

I am now also having this issue. Was there any workaround other than deploying a new queue and worker?

ajgeiss0702•7/11/24, 11:44 PM

trying to delete the worker throws "script still in use as a consumer for a queue"

ajgeiss0702•7/12/24, 1:37 AM

after manually removing the queue triggers, deleting the worker, and redeploying, the issue was fixed. Still annoying that was an issue though

33335•7/12/24, 2:38 AM

Kkian you're returning errors rather than throwing them?

daguang•7/12/24, 5:32 AM

I have try catch at the call, but I did not print the wrong log

Aajgeiss0702 I am now also having this issue. Was there any workaround other than deploying a...

Pranshu Maheshwari•7/12/24, 1:58 PM

hmm could you share your account ID & queue ID? Both are safe to share here.

ajgeiss0702•7/12/24, 1:58 PM

yes, 1m

ajgeiss0702•7/12/24, 1:59 PM

account id: f55b85c8a963663b11036975203c63c0

ajgeiss0702•7/12/24, 2:00 PM

and all of a sudden im not allowed to access the queues page in the dashboard?

ajgeiss0702•7/12/24, 2:00 PM

oh it just went through

ajgeiss0702•7/12/24, 2:00 PM

queue id: d1a94a7c4e4a4c23a6dbc74a8abbad11

WWallslide Is there a way to set a throughput limit. Like, don’t process more than 10 messa...

Pranshu Maheshwari•7/15/24, 2:25 PM

You could do it with pull based consumers: https://developers.cloudflare.com/queues/configuration/pull-consumers/

Cloudflare Docs

Cloudflare Queues - Pull consumers · Cloudflare Queues

A pull-based consumer allows you to pull from a queue over HTTP from any environment and/or programming language outside of Cloudflare Workers. A …

rcoopr•7/15/24, 9:27 PM

I think the answer to this is simply a no, but might as well ask. Is it possible with queues to process a task as soon as a single message is recieved, and dismiss the remaining messages that join the same batch?

I'm imagining the queue as a debounce of sorts, and this would be the leadingleading option for it.

Use case is something akin to logging, but latency is quite important here. Using batch timeout of ~3-5s if its relevant

Rrcoopr I think the answer to this is simply a no, but might as well ask. Is it possible...

Matt Silverlock•7/16/24, 4:57 AM

Why not set the max batch size = 1? Or just ack() all the messages you don’t care about in a batch (which means they will deleted) and only “process” what you want?

MMatt Silverlock Why not set the max batch size = 1? Or just ack() all the messages you don’t car...

rcoopr•7/16/24, 7:19 AM

I have a lot of duplicate API requests coming from separate instances of the same web page (going to some external API and don’t care about the response) and I’m looking to reduce the load on the external API and ‘dedupe’ requests via CDN/workers/queues. So a batch size of 1 would be like using the API directly I guess.

rcoopr•7/16/24, 7:20 AM

Basically I want:
Receive first api request: forward it on
Any duplicates within the next 3s, ignore

edit: ideally with POST requests, and the duplicate calls are triggered, in theory, at the same time. So when I tried simple caching with GET requests, the cache wasn't updated before duplicates came in

Both the Producer and Consumer must be the same Worker

Similar Threads

Similar Threads

Similar Threads