Is there any benefit to putting all tables (even if not associated with each other) in a single data

Is there any benefit to putting all tables (even if not associated with each other) in a single database as opposed to having multiple? All of them are used by the same worker

SSamantas5855 Is there any benefit to putting all tables (even if not associated with each oth...

Isaac McFadyen•7/18/25, 10:51 PM

D1 has a 10GB limit per DB. Also, at high request volumes you will get better performance with multiple databases versus just one. The opposite is true at low request volumes: multiple DBs might result in cold-starts if a DB isn't used for a while.

IIsaac McFadyen D1 has a 10GB limit per DB. Also, at high request volumes you will get better pe...

Samantas5855OP•7/18/25, 10:53 PM

thank you

ac•7/18/25, 11:14 PM

Are there D1 issues happening right now? We’re seeing this error, haven’t made any changes:

Error: D1_ERROR: Internal error while starting up D1 DB storage caused object to be reset

gruntlord6•7/19/25, 5:20 PM

I am also getting d1 errors today "D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset."

gruntlord6•7/19/25, 5:20 PM

I don't think I've ever seen that before, and it shouldn't be the query because If I retry it there isn't an issue

gruntlord6•7/19/25, 5:27 PM

you know what, it could be my application still, I'll keep checking into it

can•7/20/25, 6:15 AM

I'm having a similar issue right now, I can not access to my D1 instance from dashboard

lukeed•7/20/25, 2:42 PM

how are people handling global replicas and gdpr? just request.isEUCountry ? env.EUDB (no replica) : env.DBrequest.isEUCountry ? env.EUDB (no replica) : env.DB ?

James•7/20/25, 3:10 PM

I think that's the best you got right now - location hints. There's the data localisation suite (ent only), but I don't believe specific jurisdiction requirements are covered in d1 yet.

James•7/20/25, 3:17 PM

Durable Objects (and therefore DO SQLite) do support real data jurisdictions though, if that's an option for you. They just don't (yet?) support read replicas, but otherwise are probably a better option.

AAlpa Is there a query minimum requirement for read replicas to start working? I've cr...

Chaika•7/21/25, 4:25 PM

There isn't a min query requirement. Did you enable it under settings and start using the session api w/ first-unconstrained? Someone had to disable and re-enable to get it working before..

AAlpa Yes, it's enabled and i'm using first-unconstrained for all new sessions. I've a...

Chaika•7/21/25, 7:17 PM

can you /link/link your acct?

welz•7/21/25, 9:29 PM

D1_ERROR: internal error; reference = ...D1_ERROR: internal error; reference = ... - what is this?

Wwelz `D1_ERROR: internal error; reference = ...` - what is this?

Hard@Work•7/21/25, 9:30 PM

An error originating within D1 itself

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Silvan•7/22/25, 12:12 PM

the telekom special

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Chaika•7/22/25, 1:13 PM

oh interesting, yea that is normal for dtag on free plan but should still be routed to a replica for select only requests (meta.served_by_primary) even in primary region, although it won't help response times of course

KK•7/23/25, 10:15 AM

I have a requirement to stream approximately 6,000 ticker price updates per minute. I then need to support querying by ticker for all prices received today. What storage solution would you recommend?

Key-Value (KV) stores could become very expensive with 6,000 writes per minute.

I'm unsure whether D1 would be a better fit in this scenario.

I considered using the Analytics Engine, but I don't see a way to delete data older than one day. By default, it appears to retain data for 3 months, potentially growing to hundreds of millions or even billions of rows. I'm concerned about performance implications at that scale.

KKK I have a requirement to stream approximately 6,000 ticker price updates per minu...

Silvan•7/23/25, 10:45 AM

Maybe a Durable Object per ticker?

SSilvan Maybe a Durable Object per ticker?

1984 Ford Laser•7/23/25, 11:09 AM

DO per ticket per day, to keep performance consistent while allowing for any variety of data retention policy

Silvan•7/23/25, 11:10 AM

with the Actor class you even get the tracking without any additional code

KKK I have a requirement to stream approximately 6,000 ticker price updates per minu...

Larry•7/23/25, 12:07 PM

Have you looked at Cloudflare Pipelines? It's designed for high-speed data ingestion. The data goes into an R2 bucket in time slices that you can batch process.

Sylvain•7/23/25, 1:18 PM

Any plans to allow using read replicas via the D1 REST API?

LLarry Have you looked at [Cloudflare Pipelines](https://developers.cloudflare.com/pipe...

KK•7/23/25, 1:20 PM

I'm also planning to query this data once it's ingested—do you know if R2 provides low enough latency and throughput for efficient querying, or would that require an additional system layered on top?

KKK I'm also planning to query this data once it's ingested—do you know if R2 provid...

Larry•7/23/25, 1:32 PM

Yeah, you would need to process the R2 into something you could query. Unless a bunch of raw entries is boiled down into a small number of rows, I don't think D1, DOs, or KV are ideal for your situation. D1 and DOs have a 10GB limit. You could shard with DOs as Silvan suggests but that's complexity. KV is not queriable. All three are probably costly for your use. I don't know anything about it but Cloudflare Analytics might be an option. There's also hyperdrive and 3rd party storage integrations

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

Cloudflare Docs

3rd Party Integrations

Connect to third-party databases such as Supabase, Turso and PlanetScale)

KK•7/23/25, 2:05 PM

thank you very much @Larry

JJames single thread on D1 and network latency is going to kill you, you're better off ...

TeamDave•7/23/25, 2:57 PM

So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of them in a batch using drizzle, and using some results to build the last query.

Locally, this reduced my time by ~500ms. But when I go to staging & use actual D1, I'm getting a cloudflare error, "D1_ERROR: out of memory: SQLITE_NOMEM". I'm not finding a lot about that error, but it seems like I have gone too aggressive with my query combining?

Is it usually a balancing act trying to get the right number of requests with the right complexity of queries?

KKK thank you very much @Larry

Silvan•7/23/25, 3:05 PM

but yea i agree with insane numbers like that i would recommend planetscale

TTeamDave So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of...

James•7/23/25, 3:55 PM

Probably a lot of table scanning w your queries, you should analyze the sql plan for your queries and optimize your indexing.

gruntlord6•7/23/25, 8:29 PM

Does anyone know an accurate way to see your D1 Writes? the billing page doesn't line up at all with the actual data in the d1 dashboard

gruntlord6•7/23/25, 8:30 PM

its saying I have millions more writes then any of my databases account for

gruntlord6•7/23/25, 8:31 PM

Edit: ignore this part, it was just my billing period and I I was reading it wrong

gruntlord6•7/23/25, 9:21 PM

that explains the date but they still don't line up

gruntlord6•7/23/25, 9:24 PM

theres a 50 million write difference between what the dashboard shows and what my bill shows

gruntlord6•7/23/25, 9:26 PM

I also never get my billing notification when I go over but apprently thats a beta so I just assume it never will work

gruntlord6•7/23/25, 9:26 PM

just a bit frustrating to get billed with no way to know whats happening

gruntlord6•7/23/25, 9:36 PM

well I managed to track it down but I can't say I am happy that the notification didn't trigger or that I had to dig so much just to find it

gruntlord6•7/24/25, 5:40 AM

no, it didn't line up with my 2 week history in the actual databases

gruntlord6•7/24/25, 5:40 AM

thanks for that though

Silvan•7/24/25, 12:15 PM

@Community Champion :bibicat:

mrcubi•7/24/25, 12:21 PM

hi i am using cloudflare d1 service and the quote of the service i have 25 billion row read free monthly but i still get bill of that 25 billion monthly can i know why

Mmrcubi hi i am using cloudflare d1 service and the quote of the service i have 25 billi...

Silvan•7/24/25, 12:22 PM

you are on the workers paid plan right

mrcubi•7/24/25, 12:23 PM

Yes

Silvan•7/24/25, 12:23 PM

were you billed 25 dollars?

mrcubi•7/24/25, 12:24 PM

I used 38 billion in July and I got a bill for 38 billion

Silvan•7/24/25, 12:24 PM

so 38 dollar?

mrcubi•7/24/25, 12:24 PM

Yes

Is there any benefit to putting all tables (even if not associated with each other) in a single data

Similar Threads