Yes, each message is a message so both will be processed possibly concurrently if you have a busy qu

Yes, each message is a message so both will be processed possibly concurrently if you have a busy queue. You can disable concurrency though if you need to process them more controlled. See: https://developers.cloudflare.com/queues/configuration/consumer-concurrency/

AAlex Yes, each message is a message so both will be processed possibly concurrently i...

SteffTek•10/31/24, 2:27 PM

thx, yeah i have concurrency disabled currently, just hoped i overlooked something. we have a really busy queue and without concurrency it takes a while. i might add a second queue, for events where race conditions are a non-issue

SSteffTek thx, yeah i have concurrency disabled currently, just hoped i overlooked somethi...

AlexOP•10/31/24, 2:32 PM

Yeah either remove it being a problem that it runs concurrently (not sure about your use case obv. but maybe possible) or seperate the queues. You can even add more queues and "shard" them so they are still not concurrent but you have multiple queues processing. There are ways around it

SteffTek•10/31/24, 2:33 PM

we handle cash register transactions async so if anything is triggered twice we really have a problem :KEKW:

AlexOP•10/31/24, 2:33 PM

You can even pipe your concurrency challenged jobs through a durable object which can work as a gate to only allow a single operation to run.

SteffTek•10/31/24, 2:34 PM

thats, actually a really good idea

AlexOP•10/31/24, 2:35 PM

CF queues also aren't guaranteed to deliver the message once. They are at least once so I hope having the same message multiple times is not an issue actually

Original message was deleted

SteffTek•10/31/24, 2:35 PM

in germany we have to securley record every transation and transactional update, there are exactly two cloud providers for this, and one doesnt provide the other features we need... they do in fact have idempotency, but not really because they need an revision id

AAlex CF queues also aren't guaranteed to deliver the message once. They are at least ...

SteffTek•10/31/24, 2:36 PM

yeah but as long as only one consumer runs this is a non issue, because the worker filters out requests that have already run

SteffTek•10/31/24, 2:36 PM

(at least thats what i've coded lol)

AlexOP•10/31/24, 2:37 PM

That makes sense! In that case a DO is probably a very easy concurrency limiter which should allow your queue to process a lot faster and only block a little in case a concurrent request arrives. Little extra cost but should not be too bad.

SteffTek•10/31/24, 2:40 PM

it's just a bit annoying, cause our tenant isolation means we have an worker, d1, queue, etc for every tenant

SteffTek•10/31/24, 2:40 PM

but definitly doable

SteffTek•10/31/24, 2:41 PM

btw: our third party api provider has responsetimes of around 500ms per request, and we need 3-4 requests per transaction, its a pain

AlexOP•10/31/24, 2:41 PM

Fun times, you might be one of the few who will run into the script limits then

AlexOP•10/31/24, 2:41 PM

And holy sh*t that is brutal!

SteffTek•10/31/24, 2:41 PM

we'll hit the d1 limit much faster

SteffTek•10/31/24, 2:42 PM

either the 10gb one or the total storage one :D

AlexOP•10/31/24, 2:42 PM

You can have 500 workers and 50k D1 databases

You sure?

SteffTek•10/31/24, 2:42 PM

we have one client right now and multiple (big) in the pipeline, i really need to figure out to make this stuff faster

AAlex You can have 500 workers and 50k D1 databases 😛 You sure?

SteffTek•10/31/24, 2:43 PM

soft limits :P

AlexOP•10/31/24, 2:43 PM

Of course, everything is a soft limit

AlexOP•10/31/24, 2:43 PM

Fun challenges!

SteffTek•10/31/24, 2:44 PM

indeed

SteffTek•10/31/24, 2:44 PM

well, my co-workers write our desktop apps with delphi, a real challenge was to teach dem REST and JSON

SteffTek•10/31/24, 2:45 PM

4 person team, i am responsible for: backend, tenant onboarding, backoffice suit, mobile apps, infrastructure
the others: windows

SteffTek•10/31/24, 2:46 PM

and this is my third job... :KEKW:

AlexOP•10/31/24, 2:47 PM

Are you processing jobs in the queue() callback sequentially by the way?
You could do some de-duplication per-batch and concurrently process the batch

AlexOP•10/31/24, 2:48 PM

Assuming your third party can handle that load

AAlex Are you processing jobs in the queue() callback sequentially by the way? You cou...

SteffTek•10/31/24, 2:49 PM

yeah im doing preprocessing, like grouping by ids etc, thats not the issue

SteffTek•10/31/24, 2:49 PM

i only send the type of job and the id of the resource via queue

AlexOP•10/31/24, 2:50 PM

With that preprocessing are you processing those concurrentlcy? Like generate a promise foreach job and use Promise.all() to concurrently process them to speed up the batch processing?

SteffTek•10/31/24, 2:50 PM

yes

SteffTek•10/31/24, 2:50 PM

exactly ^^

AlexOP•10/31/24, 2:50 PM

Ah okay, figured that might also be a cool optimization but you already though about that

SteffTek•10/31/24, 2:52 PM

i just hope the third party api speeds up, like 100ms would be nice ^^

SteffTek•10/31/24, 2:52 PM

currently it takes up to 10 seconds for a single transaction receipt

SteffTek•10/31/24, 2:52 PM

imagine standing at the checkout and waiting 10 seconds for your receipt

AlexOP•10/31/24, 2:52 PM

They might need to learn a bit from your and starting using Workers or something because that is insane

SteffTek•10/31/24, 2:53 PM

and i need to optimize sqlite lol

SteffTek•10/31/24, 2:53 PM

but thats a whole other bag

SteffTek•10/31/24, 2:53 PM

to many sql queries

SteffTek•10/31/24, 2:54 PM

thanks for your insights ^^ have to go

AlexOP•10/31/24, 2:55 PM

np good luck!

letincho5•11/2/24, 12:12 AM

Hi, I've post a performance problem that i already have: https://stackoverflow.com/questions/79149715/cloudflare-queues-with-slow-consumer-issues

Stack Overflow

Cloudflare queues with slow consumer issues

I'm having problems with my Cloudflare queue consumers. Despite using the different configuration options: (max_batch_timeout=0, max_batch_size=1, delaySeconds=0), my workers take between 5 and 10