CF queues also aren't guaranteed to deliver the message once. They are at least once so I hope havin

CF queues also aren't guaranteed to deliver the message once. They are at least once so I hope having the same message multiple times is not an issue actually

Original message was deleted

SteffTek•10/31/24, 2:35 PM

in germany we have to securley record every transation and transactional update, there are exactly two cloud providers for this, and one doesnt provide the other features we need... they do in fact have idempotency, but not really because they need an revision id

AAlex CF queues also aren't guaranteed to deliver the message once. They are at least ...

SteffTek•10/31/24, 2:36 PM

yeah but as long as only one consumer runs this is a non issue, because the worker filters out requests that have already run

SteffTek•10/31/24, 2:36 PM

(at least thats what i've coded lol)

AlexOP•10/31/24, 2:37 PM

That makes sense! In that case a DO is probably a very easy concurrency limiter which should allow your queue to process a lot faster and only block a little in case a concurrent request arrives. Little extra cost but should not be too bad.

SteffTek•10/31/24, 2:40 PM

it's just a bit annoying, cause our tenant isolation means we have an worker, d1, queue, etc for every tenant

SteffTek•10/31/24, 2:40 PM

but definitly doable

SteffTek•10/31/24, 2:41 PM

btw: our third party api provider has responsetimes of around 500ms per request, and we need 3-4 requests per transaction, its a pain

AlexOP•10/31/24, 2:41 PM

Fun times, you might be one of the few who will run into the script limits then

AlexOP•10/31/24, 2:41 PM

And holy sh*t that is brutal!

SteffTek•10/31/24, 2:41 PM

we'll hit the d1 limit much faster

SteffTek•10/31/24, 2:42 PM

either the 10gb one or the total storage one :D

AlexOP•10/31/24, 2:42 PM

You can have 500 workers and 50k D1 databases

You sure?

SteffTek•10/31/24, 2:42 PM

we have one client right now and multiple (big) in the pipeline, i really need to figure out to make this stuff faster

AAlex You can have 500 workers and 50k D1 databases 😛 You sure?

SteffTek•10/31/24, 2:43 PM

soft limits :P

AlexOP•10/31/24, 2:43 PM

Of course, everything is a soft limit

AlexOP•10/31/24, 2:43 PM

Fun challenges!

SteffTek•10/31/24, 2:44 PM

indeed

SteffTek•10/31/24, 2:44 PM

well, my co-workers write our desktop apps with delphi, a real challenge was to teach dem REST and JSON

SteffTek•10/31/24, 2:45 PM

4 person team, i am responsible for: backend, tenant onboarding, backoffice suit, mobile apps, infrastructure
the others: windows

SteffTek•10/31/24, 2:46 PM

and this is my third job... :KEKW:

AlexOP•10/31/24, 2:47 PM

Are you processing jobs in the queue() callback sequentially by the way?
You could do some de-duplication per-batch and concurrently process the batch

AlexOP•10/31/24, 2:48 PM

Assuming your third party can handle that load

AAlex Are you processing jobs in the queue() callback sequentially by the way? You cou...

SteffTek•10/31/24, 2:49 PM

yeah im doing preprocessing, like grouping by ids etc, thats not the issue

SteffTek•10/31/24, 2:49 PM

i only send the type of job and the id of the resource via queue

AlexOP•10/31/24, 2:50 PM

With that preprocessing are you processing those concurrentlcy? Like generate a promise foreach job and use Promise.all() to concurrently process them to speed up the batch processing?

SteffTek•10/31/24, 2:50 PM

yes

SteffTek•10/31/24, 2:50 PM

exactly ^^

AlexOP•10/31/24, 2:50 PM

Ah okay, figured that might also be a cool optimization but you already though about that

SteffTek•10/31/24, 2:52 PM

i just hope the third party api speeds up, like 100ms would be nice ^^

SteffTek•10/31/24, 2:52 PM

currently it takes up to 10 seconds for a single transaction receipt

SteffTek•10/31/24, 2:52 PM

imagine standing at the checkout and waiting 10 seconds for your receipt

AlexOP•10/31/24, 2:52 PM

They might need to learn a bit from your and starting using Workers or something because that is insane

SteffTek•10/31/24, 2:53 PM

and i need to optimize sqlite lol

SteffTek•10/31/24, 2:53 PM

but thats a whole other bag

SteffTek•10/31/24, 2:53 PM

to many sql queries

SteffTek•10/31/24, 2:54 PM

thanks for your insights ^^ have to go

AlexOP•10/31/24, 2:55 PM

np good luck!

letincho5•11/2/24, 12:12 AM

Hi, I've post a performance problem that i already have: https://stackoverflow.com/questions/79149715/cloudflare-queues-with-slow-consumer-issues

Stack Overflow

Cloudflare queues with slow consumer issues

I'm having problems with my Cloudflare queue consumers. Despite using the different configuration options: (max_batch_timeout=0, max_batch_size=1, delaySeconds=0), my workers take between 5 and 10

DanGamble•11/4/24, 9:49 AM

Is there a way to consume/product queues locally when using Vite as well? I've tried having

vite

vite

and

wrangler dev

wrangler dev

running alongside each other but that doesn't seem to work

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

Hello, I’m Allie!•11/4/24, 3:05 PM

You should be able to push into a Queue, assuming your framework supports it

Hello, I’m Allie!•11/4/24, 3:05 PM

But not consume

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

James•11/4/24, 5:16 PM

when running locally i proxy the push to the queue worker's fetch which pushes to its local queue to then also consume

JJames when running locally i proxy the push to the queue worker's fetch which pushes t...

DanGamble•11/4/24, 7:45 PM

Do you have an example of this?

HHello, I’m Allie!You should be able to push into a Queue, assuming your framework supports it

DanGamble•11/4/24, 7:46 PM

Yea, I can push to it but I'm looking to consume it as well. At the moment I'm setting up a 2nd worker and using a service binding which just feels cumbersome

DDanGamble Do you have an example of this?

James•11/4/24, 7:48 PM

my main app is a pages project, and i have a separate worker for consuming the queue, on the qworker, i have a fetch which accepts the same payload as the queue send. if the origin worker is the same as the consumer worker, the queue works as expected locally, i only send to the fetch if my env is local, otherwise my pages app sends directly to the queue

DDanGamble Yea, I can push to it but I'm looking to consume it as well. At the moment I'm s...

Hello, I’m Allie!•11/4/24, 7:49 PM

Wait, so are you using a framework in Vite?

Hello, I’m Allie!•11/4/24, 7:49 PM

Or...

HHello, I’m Allie!Wait, so are you using a framework in Vite?

DanGamble•11/4/24, 7:50 PM

Yea, remix

DDanGamble Yea, remix

Hello, I’m Allie!•11/4/24, 7:50 PM

That is not currently possible, for dev anyway

CF queues also aren't guaranteed to deliver the message once. They are at least once so I hope havin

Similar Threads

Similar Threads

Similar Threads