i want to export d1 to duckdb periodically for analytical queries. does anyone have an good referenc

i want to export d1 to duckdb periodically for analytical queries. does anyone have an good reference architecture? i'm pretty worried about this mention of the docs:

A running export will block other database requests.
Source

IiLearn is following a valid SQL for execution by Cloudflare D1 .. thanks .. Reason (sh...

iLearn•7/14/25, 10:35 AM

bumping msg again for kind response .. thanks ..

Ashkan•7/14/25, 2:25 PM

Dear CF team would you please add option of Download Database SchemeDownload Database Scheme for D1 databases? why it's very important because database scheme may change in dashboard that makes some difference from original scheme that been deployed via wrangler. if there be an option to download current database scheme, then before deploying new original scheme for D1 databases we can check if there is any changes that hasn't been considered in original (local) databases scheme or not. Thank You you are the best.

AAshkan Dear CF team would you please add option of `Download Database Scheme` for D1 da...

James•7/14/25, 3:13 PM

pretty simple to download it yourself from sqlite_master

JJames pretty simple to download it yourself from sqlite_master

Ashkan•7/14/25, 3:47 PM

is it via dashboard? wrangler ? can u bring an example or a link to read about it? I haven't ever used sqlite_mastersqlite_master maybe I don't remember.

AAshkan is it via dashboard? wrangler ? can u bring an example or a link to read about i...

James•7/14/25, 3:48 PM

you gotta do it via raw sql, but all the schema data you need is already available, ask chatgpt if you're unsure how to do it in your own app context

if you're trying to manually manage schema, thats a recipe for disaster, you're better off having strict database migration cycles

JJames you gotta do it via raw sql, but all the schema data you need is already availab...

Ashkan•7/14/25, 3:50 PM

I will read about it thank you.

NNabeel Rahman why i am getting this error, can anyone help please?

Brayden•7/14/25, 6:49 PM

Do you have the code that's triggering that error available to share? TIMESTAMP isn't a data type in SQLite so I suspect it's something around your query.

NNabeel Rahman why i am getting this error, can anyone help please?

can•7/15/25, 5:01 AM

I'm getting the same error when i'm trying it. It was working just a day before :/ Could that may be related to CURRENT_TIMESTAMPCURRENT_TIMESTAMP?
These fields was already here for the last 3 months, so I assume it's related with drizzle. I installed and downgraded drizzle-zod and drizzle packages. That's maybe why.

Ddandoen i want to export d1 to duckdb periodically for analytical queries. does anyone h...

dandoenOP•7/15/25, 11:12 AM

pretty please? any guidance on just dumping it in r2 without blocking the db would also work.

Ddandoen pretty please? any guidance on just dumping it in r2 without blocking the db wou...

Brayden•7/15/25, 1:19 PM

Can't really think of a decent way of doing this without manually writing a service that queries and fetches N records at a time and then do the storing in R2 yourself in chunks to allow requests to continue between your requests. The risks there are if data changes that you've already exported then you kind of are in a bad state.

Exports on the database though will block other database requests.

BBrayden Can't really think of a decent way of doing this without manually writing a serv...

dandoenOP•7/15/25, 1:20 PM

if only there was a way – as a work around – to do point in time recovery of a d1 (replica) into a new d1, that i could then use to export and then delete aftewards. but looks like creating a d1 from a snapshot isn't supported either

TeamDave•7/15/25, 2:40 PM

Hello! I'm building out an analytics tool that makes a lot of db calls (10+ calls to get the things like total users, unique sessions, events, etc).

I'm using cloudflare pages, so locally, it's using a sqlite db and the requests can take up to 2 seconds. When I go to staging it uses the D1 db and my requests can take up to 3 minutes.

Any ideas why it would be so different? I'm thinking maybe the db connections are what's taking so long, I am making a lot of connections but don't want to spend the time trying to combine the sql if it's not going to get me down to that ~2 second mark.

TTeamDave Hello! I'm building out an analytics tool that makes a lot of db calls (10+ call...

James•7/15/25, 4:12 PM

single thread on D1 and network latency is going to kill you, you're better off batching if possible, this is one thing DO is better than D1 because you can customize how you batch queries better when you need things like shared IDs, groupig or aggregation outside of SQL. but for straight D1, you're way better off working your query into something that grabs it all in one go, especially since you don't pay for CPU on D1

IiLearn is following a valid SQL for execution by Cloudflare D1 .. thanks .. Reason (sh...

iLearn•7/17/25, 12:00 AM

Re-posting my Q .. as no answer received for 5-days ..

is following a valid SQL for execution by Cloudflare D1 .. thanks ..

Reason (shared by ChatGPT) : As of July 2025, there's no evidence that changes() is supported in D1. Its omission in docs combined with reported limitations strongly suggests it’s not available.

Worker Code Snippet with SQL execution planned :
// AIM : "to execute stmtUpdate statement ONLY if insertRes statement is success, without any JS stament in between D1 calls"
const stmts = await env.DB.batch([
env.DB.prepare(INSERT INTO users (sid) VALUES (?)).bind(sid),
env.DB.prepare(
UPDATE users
SET number_of_books = number_of_books + 1
WHERE sid = ?
AND changes() = 1
).bind(sid)
]);

SQL File : (for reference)

CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sid TEXT UNIQUE,
number_of_books INTEGER NOT NULL DEFAULT 0
);

CanRau•7/17/25, 1:02 AM

Heyho

any thought gone into the possibility of supporting pqlite in d1/do at some point?

Rob M.•7/17/25, 9:34 AM

Hi guys

What is the official flow to do migrations for a bunch of D1 dbs via REST API? As I know, D1 is about horizontal scaling and we should use rest api instead of bindings. But what about migrations in this case?

Rob M.•7/17/25, 10:03 AM

Also, can you advise a good ORM for this case?

RRob M.Hi guys 👋 What is the official flow to do migrations for a bunch of D1 dbs via ...

James•7/17/25, 3:06 PM

bindings are always better than the REST api unless you need something meta out of the API the sql won't give you

JJames bindings are always better than the REST api unless you need something meta out ...

Rob M.•7/17/25, 3:54 PM

Thanks! But afaik, you can't scale horizontally using bindings.

James•7/17/25, 3:55 PM

what do you mean by that though, thats not clear

Rob M.•7/17/25, 3:57 PM

I mean that D1 is about horizontal scaling (one db per tenant/team/user) so you need to create/delete/query lots of dbs in runtime.

Rob M.•7/17/25, 3:58 PM

You can't create bindings in runtime so the only way is using the API

James•7/17/25, 3:58 PM

the REST api has hard limits, so its pretty janky to use as an operational scale tool

for those kind of use cases, DO sql is more appropriate for horizontal scale at runtime

James•7/17/25, 4:01 PM

id avoid an ORM btw, they have oddball edge cases that break things, all to save using raw SQL, but raw SQL is way more clear and easy these days with AI

James•7/17/25, 4:03 PM

that being said, I do use drizzle-kit to manage migrations and hydrate query recordsets to JS data types based on the schema, but not to run sql.

Rob M.•7/17/25, 4:32 PM

Great points. Thanks a lot!

welz•7/17/25, 7:19 PM

Is there a D1 outage?

welz•7/17/25, 7:21 PM

From the D1 Studio

and within my worker logs:

D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr Error: D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1PreparedStatement.first (cloudflare-internal:d1-api:287:35)

D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr Error: D1_ERROR: internal error; reference = ndpgcatuhsqmvt3vuskhertr at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1PreparedStatement.first (cloudflare-internal:d1-api:287:35)

Samantas5855•7/18/25, 10:49 PM

Is there any benefit to putting all tables (even if not associated with each other) in a single database as opposed to having multiple? All of them are used by the same worker

SSamantas5855 Is there any benefit to putting all tables (even if not associated with each oth...

Isaac McFadyen•7/18/25, 10:51 PM

D1 has a 10GB limit per DB. Also, at high request volumes you will get better performance with multiple databases versus just one. The opposite is true at low request volumes: multiple DBs might result in cold-starts if a DB isn't used for a while.

IIsaac McFadyen D1 has a 10GB limit per DB. Also, at high request volumes you will get better pe...

Samantas5855•7/18/25, 10:53 PM

thank you

ac•7/18/25, 11:14 PM

Are there D1 issues happening right now? We’re seeing this error, haven’t made any changes:

Error: D1_ERROR: Internal error while starting up D1 DB storage caused object to be reset

gruntlord6•7/19/25, 5:20 PM

I am also getting d1 errors today "D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset."

gruntlord6•7/19/25, 5:20 PM

I don't think I've ever seen that before, and it shouldn't be the query because If I retry it there isn't an issue

gruntlord6•7/19/25, 5:27 PM

you know what, it could be my application still, I'll keep checking into it

can•7/20/25, 6:15 AM

I'm having a similar issue right now, I can not access to my D1 instance from dashboard

lukeed•7/20/25, 2:42 PM

how are people handling global replicas and gdpr? just request.isEUCountry ? env.EUDB (no replica) : env.DBrequest.isEUCountry ? env.EUDB (no replica) : env.DB ?

James•7/20/25, 3:10 PM

I think that's the best you got right now - location hints. There's the data localisation suite (ent only), but I don't believe specific jurisdiction requirements are covered in d1 yet.

James•7/20/25, 3:17 PM

Durable Objects (and therefore DO SQLite) do support real data jurisdictions though, if that's an option for you. They just don't (yet?) support read replicas, but otherwise are probably a better option.

AAlpa Is there a query minimum requirement for read replicas to start working? I've cr...

Chaika•7/21/25, 4:25 PM

There isn't a min query requirement. Did you enable it under settings and start using the session api w/ first-unconstrained? Someone had to disable and re-enable to get it working before..

AAlpa Yes, it's enabled and i'm using first-unconstrained for all new sessions. I've a...

Chaika•7/21/25, 7:17 PM

can you /link/link your acct?

welz•7/21/25, 9:29 PM

D1_ERROR: internal error; reference = ...D1_ERROR: internal error; reference = ... - what is this?

Wwelz `D1_ERROR: internal error; reference = ...` - what is this?

Hard@Work•7/21/25, 9:30 PM

An error originating within D1 itself

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Silvan•7/22/25, 12:12 PM

the telekom special

AAlpa Looks like this wasn't an issue with D1 itself. Apparently my provider (DTAG in ...

Chaika•7/22/25, 1:13 PM

oh interesting, yea that is normal for dtag on free plan but should still be routed to a replica for select only requests (meta.served_by_primary) even in primary region, although it won't help response times of course

i want to export d1 to duckdb periodically for analytical queries. does anyone have an good referenc

Similar Threads

Similar Threads

Similar Threads