pretty please? any guidance on just dumping it in r2 without blocking the db would also work.

Ddandoen pretty please? any guidance on just dumping it in r2 without blocking the db wou...

Brayden•7/15/25, 1:19 PM

Can't really think of a decent way of doing this without manually writing a service that queries and fetches N records at a time and then do the storing in R2 yourself in chunks to allow requests to continue between your requests. The risks there are if data changes that you've already exported then you kind of are in a bad state.

Exports on the database though will block other database requests.

BBrayden Can't really think of a decent way of doing this without manually writing a serv...

dandoenOP•7/15/25, 1:20 PM

if only there was a way – as a work around – to do point in time recovery of a d1 (replica) into a new d1, that i could then use to export and then delete aftewards. but looks like creating a d1 from a snapshot isn't supported either

TeamDave•7/15/25, 2:40 PM

Hello! I'm building out an analytics tool that makes a lot of db calls (10+ calls to get the things like total users, unique sessions, events, etc).

I'm using cloudflare pages, so locally, it's using a sqlite db and the requests can take up to 2 seconds. When I go to staging it uses the D1 db and my requests can take up to 3 minutes.

Any ideas why it would be so different? I'm thinking maybe the db connections are what's taking so long, I am making a lot of connections but don't want to spend the time trying to combine the sql if it's not going to get me down to that ~2 second mark.

TTeamDave Hello! I'm building out an analytics tool that makes a lot of db calls (10+ call...

James•7/15/25, 4:12 PM

single thread on D1 and network latency is going to kill you, you're better off batching if possible, this is one thing DO is better than D1 because you can customize how you batch queries better when you need things like shared IDs, groupig or aggregation outside of SQL. but for straight D1, you're way better off working your query into something that grabs it all in one go, especially since you don't pay for CPU on D1

IiLearn is following a valid SQL for execution by Cloudflare D1 .. thanks .. Reason (sh...

iLearn•7/17/25, 12:00 AM

Re-posting my Q .. as no answer received for 5-days ..

is following a valid SQL for execution by Cloudflare D1 .. thanks ..

Reason (shared by ChatGPT) : As of July 2025, there's no evidence that changes() is supported in D1. Its omission in docs combined with reported limitations strongly suggests it’s not available.

Worker Code Snippet with SQL execution planned :
// AIM : "to execute stmtUpdate statement ONLY if insertRes statement is success, without any JS stament in between D1 calls"
const stmts = await env.DB.batch([
env.DB.prepare(INSERT INTO users (sid) VALUES (?)).bind(sid),
env.DB.prepare(
UPDATE users
SET number_of_books = number_of_books + 1
WHERE sid = ?
AND changes() = 1
).bind(sid)
]);

SQL File : (for reference)

CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sid TEXT UNIQUE,
number_of_books INTEGER NOT NULL DEFAULT 0
);

CanRau•7/17/25, 1:02 AM

Heyho

any thought gone into the possibility of supporting pqlite in d1/do at some point?

Rob M.•7/17/25, 9:34 AM

Hi guys

What is the official flow to do migrations for a bunch of D1 dbs via REST API? As I know, D1 is about horizontal scaling and we should use rest api instead of bindings. But what about migrations in this case?

Rob M.•7/17/25, 10:03 AM

Also, can you advise a good ORM for this case?

RRob M.Hi guys 👋 What is the official flow to do migrations for a bunch of D1 dbs via ...

James•7/17/25, 3:06 PM

bindings are always better than the REST api unless you need something meta out of the API the sql won't give you

JJames bindings are always better than the REST api unless you need something meta out ...

Rob M.•7/17/25, 3:54 PM

Thanks! But afaik, you can't scale horizontally using bindings.

James•7/17/25, 3:55 PM

what do you mean by that though, thats not clear

Rob M.•7/17/25, 3:57 PM

I mean that D1 is about horizontal scaling (one db per tenant/team/user) so you need to create/delete/query lots of dbs in runtime.

Rob M.•7/17/25, 3:58 PM

You can't create bindings in runtime so the only way is using the API

James•7/17/25, 3:58 PM

the REST api has hard limits, so its pretty janky to use as an operational scale tool

for those kind of use cases, DO sql is more appropriate for horizontal scale at runtime

James•7/17/25, 4:01 PM

id avoid an ORM btw, they have oddball edge cases that break things, all to save using raw SQL, but raw SQL is way more clear and easy these days with AI

James•7/17/25, 4:03 PM

that being said, I do use drizzle-kit to manage migrations and hydrate query recordsets to JS data types based on the schema, but not to run sql.

Rob M.•7/17/25, 4:32 PM

Great points. Thanks a lot!

welz•7/17/25, 7:19 PM

Is there a D1 outage?

welz•7/17/25, 7:21 PM

From the D1 Studio

and within my worker logs:

D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr Error: D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1PreparedStatement.first (cloudflare-internal:d1-api:287:35)

D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr Error: D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1PreparedStatement.first (cloudflare-internal:d1-api:287:35)

Samantas5855•7/18/25, 10:49 PM

Is there any benefit to putting all tables (even if not associated with each other) in a single database as opposed to having multiple? All of them are used by the same worker

SSamantas5855 Is there any benefit to putting all tables (even if not associated with each oth...

Isaac McFadyen•7/18/25, 10:51 PM

D1 has a 10GB limit per DB. Also, at high request volumes you will get better performance with multiple databases versus just one. The opposite is true at low request volumes: multiple DBs might result in cold-starts if a DB isn't used for a while.

IIsaac McFadyen D1 has a 10GB limit per DB. Also, at high request volumes you will get better pe...

Samantas5855•7/18/25, 10:53 PM

thank you

ac•7/18/25, 11:14 PM

Are there D1 issues happening right now? We’re seeing this error, haven’t made any changes:

Error: D1_ERROR: Internal error while starting up D1 DB storage caused object to be reset

gruntlord6•7/19/25, 5:20 PM

I am also getting d1 errors today "D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset."

gruntlord6•7/19/25, 5:20 PM

I don't think I've ever seen that before, and it shouldn't be the query because If I retry it there isn't an issue

gruntlord6•7/19/25, 5:27 PM

you know what, it could be my application still, I'll keep checking into it

can•7/20/25, 6:15 AM

I'm having a similar issue right now, I can not access to my D1 instance from dashboard

lukeed•7/20/25, 2:42 PM

how are people handling global replicas and gdpr? just request.isEUCountry ? env.EUDB (no replica) : env.DBrequest.isEUCountry ? env.EUDB (no replica) : env.DB ?

James•7/20/25, 3:10 PM

I think that's the best you got right now - location hints. There's the data localisation suite (ent only), but I don't believe specific jurisdiction requirements are covered in d1 yet.

James•7/20/25, 3:17 PM

Durable Objects (and therefore DO SQLite) do support real data jurisdictions though, if that's an option for you. They just don't (yet?) support read replicas, but otherwise are probably a better option.

AAlpa Is there a query minimum requirement for read replicas to start working? I've cr...

Chaika•7/21/25, 4:25 PM

There isn't a min query requirement. Did you enable it under settings and start using the session api w/ first-unconstrained? Someone had to disable and re-enable to get it working before..

AAlpa Yes, it's enabled and i'm using first-unconstrained for all new sessions. I've a...

Chaika•7/21/25, 7:17 PM

can you /link/link your acct?

welz•7/21/25, 9:29 PM

D1_ERROR: internal error; reference = ...D1_ERROR: internal error; reference = ... - what is this?

Wwelz `D1_ERROR: internal error; reference = ...` - what is this?

Hard@Work•7/21/25, 9:30 PM

An error originating within D1 itself

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Silvan•7/22/25, 12:12 PM

the telekom special

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Chaika•7/22/25, 1:13 PM

oh interesting, yea that is normal for dtag on free plan but should still be routed to a replica for select only requests (meta.served_by_primary) even in primary region, although it won't help response times of course

KK•7/23/25, 10:15 AM

I have a requirement to stream approximately 6,000 ticker price updates per minute. I then need to support querying by ticker for all prices received today. What storage solution would you recommend?

Key-Value (KV) stores could become very expensive with 6,000 writes per minute.

I'm unsure whether D1 would be a better fit in this scenario.

I considered using the Analytics Engine, but I don't see a way to delete data older than one day. By default, it appears to retain data for 3 months, potentially growing to hundreds of millions or even billions of rows. I'm concerned about performance implications at that scale.

KKK I have a requirement to stream approximately 6,000 ticker price updates per minu...

Silvan•7/23/25, 10:45 AM

Maybe a Durable Object per ticker?

SSilvan Maybe a Durable Object per ticker?

1984 Ford Laser•7/23/25, 11:09 AM

DO per ticket per day, to keep performance consistent while allowing for any variety of data retention policy

Silvan•7/23/25, 11:10 AM

with the Actor class you even get the tracking without any additional code

KKK I have a requirement to stream approximately 6,000 ticker price updates per minu...

Larry•7/23/25, 12:07 PM

Have you looked at Cloudflare Pipelines? It's designed for high-speed data ingestion. The data goes into an R2 bucket in time slices that you can batch process.

Sylvain•7/23/25, 1:18 PM

Any plans to allow using read replicas via the D1 REST API?

LLarry Have you looked at [Cloudflare Pipelines](https://developers.cloudflare.com/pipe...

KK•7/23/25, 1:20 PM

I'm also planning to query this data once it's ingested—do you know if R2 provides low enough latency and throughput for efficient querying, or would that require an additional system layered on top?

KKK I'm also planning to query this data once it's ingested—do you know if R2 provid...

Larry•7/23/25, 1:32 PM

Yeah, you would need to process the R2 into something you could query. Unless a bunch of raw entries is boiled down into a small number of rows, I don't think D1, DOs, or KV are ideal for your situation. D1 and DOs have a 10GB limit. You could shard with DOs as Silvan suggests but that's complexity. KV is not queriable. All three are probably costly for your use. I don't know anything about it but Cloudflare Analytics might be an option. There's also hyperdrive and 3rd party storage integrations

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

Cloudflare Docs

3rd Party Integrations

Connect to third-party databases such as Supabase, Turso and PlanetScale)

KK•7/23/25, 2:05 PM

thank you very much @Larry

JJames single thread on D1 and network latency is going to kill you, you're better off ...

TeamDave•7/23/25, 2:57 PM

So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of them in a batch using drizzle, and using some results to build the last query.

Locally, this reduced my time by ~500ms. But when I go to staging & use actual D1, I'm getting a cloudflare error, "D1_ERROR: out of memory: SQLITE_NOMEM". I'm not finding a lot about that error, but it seems like I have gone too aggressive with my query combining?

Is it usually a balancing act trying to get the right number of requests with the right complexity of queries?

pretty please? any guidance on just dumping it in r2 without blocking the db would also work.

Similar Threads

Similar Threads

Similar Threads