Hi guys 👋 What is the official flow to do migrations for a bunch of D1 dbs via REST API? As I know,

Hi guys

What is the official flow to do migrations for a bunch of D1 dbs via REST API? As I know, D1 is about horizontal scaling and we should use rest api instead of bindings. But what about migrations in this case?

Rob M.OP•7/17/25, 10:03 AM

Also, can you advise a good ORM for this case?

RRob M.Hi guys 👋 What is the official flow to do migrations for a bunch of D1 dbs via ...

James•7/17/25, 3:06 PM

bindings are always better than the REST api unless you need something meta out of the API the sql won't give you

JJames bindings are always better than the REST api unless you need something meta out ...

Rob M.OP•7/17/25, 3:54 PM

Thanks! But afaik, you can't scale horizontally using bindings.

James•7/17/25, 3:55 PM

what do you mean by that though, thats not clear

Rob M.OP•7/17/25, 3:57 PM

I mean that D1 is about horizontal scaling (one db per tenant/team/user) so you need to create/delete/query lots of dbs in runtime.

Rob M.OP•7/17/25, 3:58 PM

You can't create bindings in runtime so the only way is using the API

James•7/17/25, 3:58 PM

the REST api has hard limits, so its pretty janky to use as an operational scale tool

for those kind of use cases, DO sql is more appropriate for horizontal scale at runtime

James•7/17/25, 4:01 PM

id avoid an ORM btw, they have oddball edge cases that break things, all to save using raw SQL, but raw SQL is way more clear and easy these days with AI

James•7/17/25, 4:03 PM

that being said, I do use drizzle-kit to manage migrations and hydrate query recordsets to JS data types based on the schema, but not to run sql.

Rob M.OP•7/17/25, 4:32 PM

Great points. Thanks a lot!

welz•7/17/25, 7:19 PM

Is there a D1 outage?

welz•7/17/25, 7:21 PM

From the D1 Studio

and within my worker logs:

D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr Error: D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1PreparedStatement.first (cloudflare-internal:d1-api:287:35)

D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr Error: D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1PreparedStatement.first (cloudflare-internal:d1-api:287:35)

Samantas5855•7/18/25, 10:49 PM

Is there any benefit to putting all tables (even if not associated with each other) in a single database as opposed to having multiple? All of them are used by the same worker

SSamantas5855 Is there any benefit to putting all tables (even if not associated with each oth...

Isaac McFadyen•7/18/25, 10:51 PM

D1 has a 10GB limit per DB. Also, at high request volumes you will get better performance with multiple databases versus just one. The opposite is true at low request volumes: multiple DBs might result in cold-starts if a DB isn't used for a while.

IIsaac McFadyen D1 has a 10GB limit per DB. Also, at high request volumes you will get better pe...

Samantas5855•7/18/25, 10:53 PM

thank you

ac•7/18/25, 11:14 PM

Are there D1 issues happening right now? We’re seeing this error, haven’t made any changes:

Error: D1_ERROR: Internal error while starting up D1 DB storage caused object to be reset

gruntlord6•7/19/25, 5:20 PM

I am also getting d1 errors today "D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset."

gruntlord6•7/19/25, 5:20 PM

I don't think I've ever seen that before, and it shouldn't be the query because If I retry it there isn't an issue

gruntlord6•7/19/25, 5:27 PM

you know what, it could be my application still, I'll keep checking into it

can•7/20/25, 6:15 AM

I'm having a similar issue right now, I can not access to my D1 instance from dashboard

lukeed•7/20/25, 2:42 PM

how are people handling global replicas and gdpr? just request.isEUCountry ? env.EUDB (no replica) : env.DBrequest.isEUCountry ? env.EUDB (no replica) : env.DB ?

James•7/20/25, 3:10 PM

I think that's the best you got right now - location hints. There's the data localisation suite (ent only), but I don't believe specific jurisdiction requirements are covered in d1 yet.

James•7/20/25, 3:17 PM

Durable Objects (and therefore DO SQLite) do support real data jurisdictions though, if that's an option for you. They just don't (yet?) support read replicas, but otherwise are probably a better option.

AAlpa Is there a query minimum requirement for read replicas to start working? I've cr...

Chaika•7/21/25, 4:25 PM

There isn't a min query requirement. Did you enable it under settings and start using the session api w/ first-unconstrained? Someone had to disable and re-enable to get it working before..

AAlpa Yes, it's enabled and i'm using first-unconstrained for all new sessions. I've a...

Chaika•7/21/25, 7:17 PM

can you /link/link your acct?

welz•7/21/25, 9:29 PM

D1_ERROR: internal error; reference = ...D1_ERROR: internal error; reference = ... - what is this?

Wwelz `D1_ERROR: internal error; reference = ...` - what is this?

Hard@Work•7/21/25, 9:30 PM

An error originating within D1 itself

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Silvan•7/22/25, 12:12 PM

the telekom special

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Chaika•7/22/25, 1:13 PM

oh interesting, yea that is normal for dtag on free plan but should still be routed to a replica for select only requests (meta.served_by_primary) even in primary region, although it won't help response times of course

KK•7/23/25, 10:15 AM

I have a requirement to stream approximately 6,000 ticker price updates per minute. I then need to support querying by ticker for all prices received today. What storage solution would you recommend?

Key-Value (KV) stores could become very expensive with 6,000 writes per minute.

I'm unsure whether D1 would be a better fit in this scenario.

I considered using the Analytics Engine, but I don't see a way to delete data older than one day. By default, it appears to retain data for 3 months, potentially growing to hundreds of millions or even billions of rows. I'm concerned about performance implications at that scale.

KKK I have a requirement to stream approximately 6,000 ticker price updates per minu...

Silvan•7/23/25, 10:45 AM

Maybe a Durable Object per ticker?

SSilvan Maybe a Durable Object per ticker?

1984 Ford Laser•7/23/25, 11:09 AM

DO per ticket per day, to keep performance consistent while allowing for any variety of data retention policy

Silvan•7/23/25, 11:10 AM

with the Actor class you even get the tracking without any additional code

KKK I have a requirement to stream approximately 6,000 ticker price updates per minu...

Larry•7/23/25, 12:07 PM

Have you looked at Cloudflare Pipelines? It's designed for high-speed data ingestion. The data goes into an R2 bucket in time slices that you can batch process.

Sylvain•7/23/25, 1:18 PM

Any plans to allow using read replicas via the D1 REST API?

LLarry Have you looked at [Cloudflare Pipelines](https://developers.cloudflare.com/pipe...

KK•7/23/25, 1:20 PM

I'm also planning to query this data once it's ingested—do you know if R2 provides low enough latency and throughput for efficient querying, or would that require an additional system layered on top?

KKK I'm also planning to query this data once it's ingested—do you know if R2 provid...

Larry•7/23/25, 1:32 PM

Yeah, you would need to process the R2 into something you could query. Unless a bunch of raw entries is boiled down into a small number of rows, I don't think D1, DOs, or KV are ideal for your situation. D1 and DOs have a 10GB limit. You could shard with DOs as Silvan suggests but that's complexity. KV is not queriable. All three are probably costly for your use. I don't know anything about it but Cloudflare Analytics might be an option. There's also hyperdrive and 3rd party storage integrations

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

Cloudflare Docs

3rd Party Integrations

Connect to third-party databases such as Supabase, Turso and PlanetScale)

KK•7/23/25, 2:05 PM

thank you very much @Larry

JJames single thread on D1 and network latency is going to kill you, you're better off ...

TeamDave•7/23/25, 2:57 PM

So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of them in a batch using drizzle, and using some results to build the last query.

Locally, this reduced my time by ~500ms. But when I go to staging & use actual D1, I'm getting a cloudflare error, "D1_ERROR: out of memory: SQLITE_NOMEM". I'm not finding a lot about that error, but it seems like I have gone too aggressive with my query combining?

Is it usually a balancing act trying to get the right number of requests with the right complexity of queries?

KKK thank you very much @Larry

Silvan•7/23/25, 3:05 PM

but yea i agree with insane numbers like that i would recommend planetscale

TTeamDave So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of...

James•7/23/25, 3:55 PM

Probably a lot of table scanning w your queries, you should analyze the sql plan for your queries and optimize your indexing.

gruntlord6•7/23/25, 8:29 PM

Does anyone know an accurate way to see your D1 Writes? the billing page doesn't line up at all with the actual data in the d1 dashboard

gruntlord6•7/23/25, 8:30 PM

its saying I have millions more writes then any of my databases account for

gruntlord6•7/23/25, 8:31 PM

Edit: ignore this part, it was just my billing period and I I was reading it wrong

gruntlord6•7/23/25, 9:21 PM

that explains the date but they still don't line up

gruntlord6•7/23/25, 9:24 PM

theres a 50 million write difference between what the dashboard shows and what my bill shows

Hi guys 👋 What is the official flow to do migrations for a bunch of D1 dbs via REST API? As I know,

Similar Threads

Similar Threads

Similar Threads