thx, yeah i have concurrency disabled currently, just hoped i overlooked something. we have a really

thx, yeah i have concurrency disabled currently, just hoped i overlooked something. we have a really busy queue and without concurrency it takes a while. i might add a second queue, for events where race conditions are a non-issue

SSteffTek thx, yeah i have concurrency disabled currently, just hoped i overlooked somethi...

Alex•10/31/24, 2:32 PM

Yeah either remove it being a problem that it runs concurrently (not sure about your use case obv. but maybe possible) or seperate the queues. You can even add more queues and "shard" them so they are still not concurrent but you have multiple queues processing. There are ways around it

SteffTekOP•10/31/24, 2:33 PM

we handle cash register transactions async so if anything is triggered twice we really have a problem :KEKW:

Alex•10/31/24, 2:33 PM

You can even pipe your concurrency challenged jobs through a durable object which can work as a gate to only allow a single operation to run.

SteffTekOP•10/31/24, 2:34 PM

thats, actually a really good idea

Alex•10/31/24, 2:35 PM

CF queues also aren't guaranteed to deliver the message once. They are at least once so I hope having the same message multiple times is not an issue actually

Original message was deleted

SteffTekOP•10/31/24, 2:35 PM

in germany we have to securley record every transation and transactional update, there are exactly two cloud providers for this, and one doesnt provide the other features we need... they do in fact have idempotency, but not really because they need an revision id

AAlex CF queues also aren't guaranteed to deliver the message once. They are at least ...

SteffTekOP•10/31/24, 2:36 PM

yeah but as long as only one consumer runs this is a non issue, because the worker filters out requests that have already run

SteffTekOP•10/31/24, 2:36 PM

(at least thats what i've coded lol)

Alex•10/31/24, 2:37 PM

That makes sense! In that case a DO is probably a very easy concurrency limiter which should allow your queue to process a lot faster and only block a little in case a concurrent request arrives. Little extra cost but should not be too bad.

SteffTekOP•10/31/24, 2:40 PM

it's just a bit annoying, cause our tenant isolation means we have an worker, d1, queue, etc for every tenant

SteffTekOP•10/31/24, 2:40 PM

but definitly doable

SteffTekOP•10/31/24, 2:41 PM

btw: our third party api provider has responsetimes of around 500ms per request, and we need 3-4 requests per transaction, its a pain

Alex•10/31/24, 2:41 PM

Fun times, you might be one of the few who will run into the script limits then

Alex•10/31/24, 2:41 PM

And holy sh*t that is brutal!

SteffTekOP•10/31/24, 2:41 PM

we'll hit the d1 limit much faster

SteffTekOP•10/31/24, 2:42 PM

either the 10gb one or the total storage one :D

Alex•10/31/24, 2:42 PM

You can have 500 workers and 50k D1 databases

You sure?

SteffTekOP•10/31/24, 2:42 PM

we have one client right now and multiple (big) in the pipeline, i really need to figure out to make this stuff faster

AAlex You can have 500 workers and 50k D1 databases 😛 You sure?

SteffTekOP•10/31/24, 2:43 PM

soft limits :P

Alex•10/31/24, 2:43 PM

Of course, everything is a soft limit

Alex•10/31/24, 2:43 PM

Fun challenges!

SteffTekOP•10/31/24, 2:44 PM

indeed

SteffTekOP•10/31/24, 2:44 PM

well, my co-workers write our desktop apps with delphi, a real challenge was to teach dem REST and JSON

SteffTekOP•10/31/24, 2:45 PM

4 person team, i am responsible for: backend, tenant onboarding, backoffice suit, mobile apps, infrastructure
the others: windows

SteffTekOP•10/31/24, 2:46 PM

and this is my third job... :KEKW:

Alex•10/31/24, 2:47 PM

Are you processing jobs in the queue() callback sequentially by the way?
You could do some de-duplication per-batch and concurrently process the batch

Alex•10/31/24, 2:48 PM

Assuming your third party can handle that load

AAlex Are you processing jobs in the queue() callback sequentially by the way? You cou...

SteffTekOP•10/31/24, 2:49 PM

yeah im doing preprocessing, like grouping by ids etc, thats not the issue

SteffTekOP•10/31/24, 2:49 PM

i only send the type of job and the id of the resource via queue

Alex•10/31/24, 2:50 PM

With that preprocessing are you processing those concurrentlcy? Like generate a promise foreach job and use Promise.all() to concurrently process them to speed up the batch processing?

SteffTekOP•10/31/24, 2:50 PM

yes

SteffTekOP•10/31/24, 2:50 PM

exactly ^^

Alex•10/31/24, 2:50 PM

Ah okay, figured that might also be a cool optimization but you already though about that

SteffTekOP•10/31/24, 2:52 PM

i just hope the third party api speeds up, like 100ms would be nice ^^

SteffTekOP•10/31/24, 2:52 PM

currently it takes up to 10 seconds for a single transaction receipt

SteffTekOP•10/31/24, 2:52 PM

imagine standing at the checkout and waiting 10 seconds for your receipt

Alex•10/31/24, 2:52 PM

They might need to learn a bit from your and starting using Workers or something because that is insane

SteffTekOP•10/31/24, 2:53 PM

and i need to optimize sqlite lol

SteffTekOP•10/31/24, 2:53 PM

but thats a whole other bag

SteffTekOP•10/31/24, 2:53 PM

to many sql queries

SteffTekOP•10/31/24, 2:54 PM

thanks for your insights ^^ have to go

Alex•10/31/24, 2:55 PM

np good luck!

letincho5•11/2/24, 12:12 AM

Hi, I've post a performance problem that i already have: https://stackoverflow.com/questions/79149715/cloudflare-queues-with-slow-consumer-issues

Stack Overflow

Cloudflare queues with slow consumer issues

I'm having problems with my Cloudflare queue consumers. Despite using the different configuration options: (max_batch_timeout=0, max_batch_size=1, delaySeconds=0), my workers take between 5 and 10

DanGamble•11/4/24, 9:49 AM

Is there a way to consume/product queues locally when using Vite as well? I've tried having vitevite and

wrangler dev

wrangler dev

running alongside each other but that doesn't seem to work

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

Hello, I’m Allie!•11/4/24, 3:05 PM

You should be able to push into a Queue, assuming your framework supports it

Hello, I’m Allie!•11/4/24, 3:05 PM

But not consume

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

James•11/4/24, 5:16 PM

when running locally i proxy the push to the queue worker's fetch which pushes to its local queue to then also consume

JJames when running locally i proxy the push to the queue worker's fetch which pushes t...

DanGamble•11/4/24, 7:45 PM

Do you have an example of this?

HHello, I’m Allie!You should be able to push into a Queue, assuming your framework supports it

DanGamble•11/4/24, 7:46 PM

Yea, I can push to it but I'm looking to consume it as well. At the moment I'm setting up a 2nd worker and using a service binding which just feels cumbersome

thx, yeah i have concurrency disabled currently, just hoped i overlooked something. we have a really

Similar Threads

Similar Threads

Similar Threads