You can have 500 workers and 50k D1 databases 😛 You sure? - Cloudflare Developers

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

SteffTek

S

SteffTek•10/31/24, 2:42 PM

we have one client right now and multiple (big) in the pipeline, i really need to figure out to make this stuff faster

Alex

AAlex You can have 500 workers and 50k D1 databases 😛 You sure?

SteffTek

S

SteffTek•10/31/24, 2:43 PM

soft limits :P

Alex

A

AlexOP•10/31/24, 2:43 PM

Of course, everything is a soft limit

Alex

A

AlexOP•10/31/24, 2:43 PM

Fun challenges!

SteffTek

S

SteffTek•10/31/24, 2:44 PM

indeed

SteffTek

S

SteffTek•10/31/24, 2:44 PM

well, my co-workers write our desktop apps with delphi, a real challenge was to teach dem REST and JSON

SteffTek

S

SteffTek•10/31/24, 2:45 PM

4 person team, i am responsible for: backend, tenant onboarding, backoffice suit, mobile apps, infrastructure
the others: windows

SteffTek

S

SteffTek•10/31/24, 2:46 PM

and this is my third job... :KEKW:

Alex

A

AlexOP•10/31/24, 2:47 PM

Are you processing jobs in the queue() callback sequentially by the way?
You could do some de-duplication per-batch and concurrently process the batch

Alex

A

AlexOP•10/31/24, 2:48 PM

Assuming your third party can handle that load

Alex

AAlex Are you processing jobs in the queue() callback sequentially by the way? You cou...

SteffTek

S

SteffTek•10/31/24, 2:49 PM

yeah im doing preprocessing, like grouping by ids etc, thats not the issue

SteffTek

S

SteffTek•10/31/24, 2:49 PM

i only send the type of job and the id of the resource via queue

Alex

A

AlexOP•10/31/24, 2:50 PM

With that preprocessing are you processing those concurrentlcy? Like generate a promise foreach job and use Promise.all() to concurrently process them to speed up the batch processing?

SteffTek

S

SteffTek•10/31/24, 2:50 PM

yes

SteffTek

S

SteffTek•10/31/24, 2:50 PM

exactly ^^

Alex

A

AlexOP•10/31/24, 2:50 PM

Ah okay, figured that might also be a cool optimization but you already though about that

SteffTek

S

SteffTek•10/31/24, 2:52 PM

i just hope the third party api speeds up, like 100ms would be nice ^^

SteffTek

S

SteffTek•10/31/24, 2:52 PM

currently it takes up to 10 seconds for a single transaction receipt

SteffTek

S

SteffTek•10/31/24, 2:52 PM

imagine standing at the checkout and waiting 10 seconds for your receipt

Alex

A

AlexOP•10/31/24, 2:52 PM

They might need to learn a bit from your and starting using Workers or something because that is insane

SteffTek

S

SteffTek•10/31/24, 2:53 PM

and i need to optimize sqlite lol

SteffTek

S

SteffTek•10/31/24, 2:53 PM

but thats a whole other bag

SteffTek

S

SteffTek•10/31/24, 2:53 PM

to many sql queries

SteffTek

S

SteffTek•10/31/24, 2:54 PM

thanks for your insights ^^ have to go

Alex

A

AlexOP•10/31/24, 2:55 PM

np good luck!

letincho5

L

letincho5•11/2/24, 12:12 AM

Hi, I've post a performance problem that i already have: https://stackoverflow.com/questions/79149715/cloudflare-queues-with-slow-consumer-issues

Stack Overflow

Cloudflare queues with slow consumer issues

I'm having problems with my Cloudflare queue consumers. Despite using the different configuration options: (max_batch_timeout=0, max_batch_size=1, delaySeconds=0), my workers take between 5 and 10

DanGamble

D

DanGamble•11/4/24, 9:49 AM

Is there a way to consume/product queues locally when using Vite as well? I've tried having

vite

vite

and

wrangler dev

wrangler dev

running alongside each other but that doesn't seem to work

DanGamble

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

Hello, I’m Allie!

H

Hello, I’m Allie!•11/4/24, 3:05 PM

You should be able to push into a Queue, assuming your framework supports it

Hello, I’m Allie!

H

Hello, I’m Allie!•11/4/24, 3:05 PM

But not consume

DanGamble

DDanGamble Is there a way to consume/product queues locally when using Vite as well? I've t...

James

J

James•11/4/24, 5:16 PM

when running locally i proxy the push to the queue worker's fetch which pushes to its local queue to then also consume

James

JJames when running locally i proxy the push to the queue worker's fetch which pushes t...

DanGamble

D

DanGamble•11/4/24, 7:45 PM

Do you have an example of this?

Hello, I’m Allie!

HHello, I’m Allie!You should be able to push into a Queue, assuming your framework supports it

DanGamble

D

DanGamble•11/4/24, 7:46 PM

Yea, I can push to it but I'm looking to consume it as well. At the moment I'm setting up a 2nd worker and using a service binding which just feels cumbersome

DanGamble

DDanGamble Do you have an example of this?

James

J

James•11/4/24, 7:48 PM

my main app is a pages project, and i have a separate worker for consuming the queue, on the qworker, i have a fetch which accepts the same payload as the queue send. if the origin worker is the same as the consumer worker, the queue works as expected locally, i only send to the fetch if my env is local, otherwise my pages app sends directly to the queue

DanGamble

DDanGamble Yea, I can push to it but I'm looking to consume it as well. At the moment I'm s...

Hello, I’m Allie!

H

Hello, I’m Allie!•11/4/24, 7:49 PM

Wait, so are you using a framework in Vite?

Hello, I’m Allie!

H

Hello, I’m Allie!•11/4/24, 7:49 PM

Or...

Hello, I’m Allie!

HHello, I’m Allie!Wait, so are you using a framework in Vite?

DanGamble

D

DanGamble•11/4/24, 7:50 PM

Yea, remix

DanGamble

DDanGamble Yea, remix

Hello, I’m Allie!

H

Hello, I’m Allie!•11/4/24, 7:50 PM

That is not currently possible, for dev anyway

Hello, I’m Allie!

H

Hello, I’m Allie!•11/4/24, 7:51 PM

The Frameworks team is working on a way to allow Vite-based dev servers to run in

workerd

workerd

, which would also allow them to expose non-HTTP event handlers(like Queues)

DanGamble

D

DanGamble•11/4/24, 7:51 PM

I thought as much. I imagine that's this: https://github.com/flarelabs-net/vite-plugin-cloudflare. If so, can't wait for it!

DoggeSlapper

D

DoggeSlapper•11/7/24, 4:26 PM

Queue wrangler dev concurrency is 1 right?

Baltazar

B

Baltazar•11/7/24, 5:02 PM

does someone have a code example with pull consumers?

oof2win2

O

oof2win2•11/7/24, 7:21 PM

is there a way with which i could limit the processing rate of a queue besides with concurrency + batch size? like i want to limit a certain number of messages / h

oof2win2

O

oof2win2•11/7/24, 7:22 PM

actually nvm i can do this with D1 and a queue table (even though kinda dumb, it works tho)

oof2win2

Ooof2win2 is there a way with which i could limit the processing rate of a queue besides w...

DoggeSlapper

D

DoggeSlapper•11/7/24, 7:33 PM

u can using pull style consumer

oof2win2

O

oof2win2•11/7/24, 7:33 PM

i'm using workers so that doesn't really make sense here

oof2win2

O

oof2win2•11/7/24, 7:34 PM

easier to build my "own" queue via d1 + cron

DoggeSlapper

DDoggeSlapper Queue wrangler dev concurrency is 1 right?

DoggeSlapper

D

DoggeSlapper•11/7/24, 7:40 PM

Is there a way to run multiple consumers locally?

DoggeSlapper

D

DoggeSlapper•11/7/24, 7:47 PM

I mean for concurrency not 2 diffrent consumers on same queue, or there is no way in local to have concurrency?

Sam

S

Sam•11/11/24, 2:58 PM

Is there anyway to increase concurrency beyond 250 workers?

I had 900,000 messages to queue last night and it did work but it took over 10 hours to complete all of the workload.