Cloudflare Developers•3y ago

Sounds good! And just to make sure, Consumers are guaranteed to not share resources with Producers,

Sounds good! And just to make sure, Consumers are guaranteed to not share resources with Producers, right?

Hello, I’m Allie!OP•3/25/23, 11:56 AM

At least if it isn’t the same script(like a DO being the producer)?

HHello, I’m Allie!Sounds good! And just to make sure, Consumers are guaranteed to not share resour...

Matt Silverlock•3/25/23, 11:58 AM

Your producer(s) can be any script. Even if your consumer is defined in the same script, they are invoked entirely separately

Matt Silverlock•3/25/23, 11:58 AM

No different from fetch() and scheduled() in the same script.

MMatt Silverlock Your producer(s) can be any script. Even if your consumer is defined in the same...

Hello, I’m Allie!OP•3/25/23, 12:00 PM

No, I meant like, can they end up sharing a thread/memory? I have a task that is quite resource intensive, so I don’t want my Producers/Consumers to run in the same space and slow down/run out of memory

HHello, I’m Allie!No, I meant like, can they end up sharing a thread/memory? I have a task that is...

Matt Silverlock•3/25/23, 12:03 PM

No, they don’t share the same invocation (isolate). There is no relation between the two. The “producer” isn’t really a thing to model - it’s just an API.

Matt Silverlock•3/25/23, 12:04 PM

a producer isn’t a producer: it’s a fetch or scheduled Worker calling an API (send/sendBatch)

Hello, I’m Allie!OP•3/25/23, 12:04 PM

Oh, sorry. By producer, I meant the isolate calling send/sendBatch(which I guess isn’t actually called the producer)

cb•3/26/23, 3:34 AM

Is there any progress on accessing a Queue from outside of Cloudflare?
Similar to https://docs.upstash.com/qstash

Getting Started | Upstash: Documentation

Introduction

Ccb Is there any progress on accessing a Queue from outside of Cloudflare? Similar t...

zegevlier•3/26/23, 9:02 AM

Not at the moment (I believe), but that is something that is planned as per queues

Zzegevlier Not at the moment (I believe), but that is something that is planned as per http...

Matt Silverlock•3/26/23, 10:10 AM

This is where we stand right now, yep.

Ccb Is there any progress on accessing a Queue from outside of Cloudflare? Similar t...

Matt Silverlock•3/26/23, 10:11 AM

Are you looking for push (write to) or pull (consume from?). HTTP Pull -- being able to pull from any HTTP-speaking client -- is being worked on.

cb•3/26/23, 10:12 AM

Thanks! Yes @Matt CF Worker Push to Queue -> HTTP Pull to process queue

Aakira.eth any miniflare functions for reading queues and adding data to queues

Emilio•3/28/23, 3:53 PM

Miniflare has rudimentary support for Queues, you can publish and consume messages. Expect full-featured and production-like support in Miniflare v3.

Aakira.eth If I am testing just a consumer locally any way to add messages to the queues?

Emilio•3/29/23, 2:51 PM

Yes miniflare will do that for you automatically!

Emilio•3/29/23, 3:26 PM

You need to have your producer and consumer configured in your

wrangler.toml

wrangler.toml

and start the application in local mode using

wrangler dev --local

wrangler dev --local

kian•3/30/23, 9:22 AM

Using

--env production

--env production

would require that your bindings are under

env.production

env.production

- i.e

[[env.production.queues.producers]]

[[env.production.queues.producers]]

asp•3/30/23, 2:30 PM

How long can a consumer take to process a batch of messages? Edit: 15 minutes according to the docs

Samy•4/1/23, 9:06 PM

Are the consumer and the parent worker sharing their CPU / memory (just like a parent worker and a service binding do)? or can queues be used to spread intensive operations accros multiple worker instances?

Unsmart•4/1/23, 9:20 PM

the consumer doesnt even run at the same time as the worker that inserts an item

Unsmart•4/1/23, 9:20 PM

Queues will process tasks async not sync

Unsmart•4/1/23, 9:20 PM

If you want something synchronous you would need to use service bindings

Samy•4/1/23, 9:22 PM

Yes, it makes sense, so it implies that resources are not shared, even between batches?
If I send 1000 messages with

max_batch_size: 10

max_batch_size: 10

, will I get 100 different instances with their own 128MB / CPU limit ?

Unsmart•4/1/23, 9:25 PM

Not sure but you could limit concurrency to 1 and then it will process them on a worker with full limits (possibly full limits at least). Its possible just any worker instance at all for the same account on the same server will share resources.

Samy•4/1/23, 9:33 PM

Thanks!

Matt Silverlock•4/1/23, 10:29 PM

Each batch is an invocation, just like an incoming HTTP request is @samypesse

Matt Silverlock•4/1/23, 10:31 PM

Concurrency on or off doesn’t change that. Leave concurrency on if message latency is important vs cap it if you have upstream endpoints with rate limits.

ZZiga Zajc Can queues also works for external servers? For example worker will try to `POS...

Matt Silverlock•4/2/23, 9:32 AM

You can write to a queue however you’d like! The tutorial is just a Worker that accepts a HTTP request and writes some request data to the queue.

Sounds good! And just to make sure, Consumers are guaranteed to not share resources with Producers,

Similar Threads

Similar Threads

Similar Threads