Hi! On a internal computer of our company, we would like to export our production D1 database every

Hi! On a internal computer of our company, we would like to export our production D1 database every 10 minutes ( from the CLI wrangler d1 export ). Is someone from the CF staff can confirm us it's tolerated and this will not be blocked or against the CF policies please?

GGuillaumeLakano Hi! On a internal computer of our company, we would like to export our productio...

lambrospetrou•8/22/25, 4:06 PM

There is no policy against doing that, but keep in mind that whle exporting your database is blocked from serving any queries, so doing it every 10 minutes, especially if it's more than a few hundred MBs, it's not going to be a nice experience to your users.

Original message was deleted

lambrospetrou•8/22/25, 5:31 PM

Not at the moment. Hopefully we will get to refactoring the whole export flow soon, and that won't be necessary.

Larry•8/22/25, 5:31 PM

With read-replicas, it should be a lot easier. Detach a replica and export that.

GGuillaumeLakano Hi! On a internal computer of our company, we would like to export our productio...

gruntlord6•8/22/25, 10:32 PM

Every 10 minutes seems excessive since you can just read the database directly or use time travel for restores between your backup periods. Is there something in the use case that requires that frequency?

Ggruntlord6 have a worker dump the tables

Miku•8/23/25, 11:13 AM

use worker to export?

MMiku use worker to export?

gruntlord6•8/23/25, 1:20 PM

workers can read table entries so you can just have it read the tables and save it to r2

Advany•8/25/25, 9:08 AM

how can I protect d1 that it can't be access using wrangler? (I want to prevent claude code from deleting data or dropping tables etc)

Devoto•8/25/25, 10:45 AM

Hey there, just wondering, for highly relational data, how does 10GB size limit make sense? How do production apps go around that?

DDevoto Hey there, just wondering, for highly relational data, how does 10GB size limit ...

Larry•8/25/25, 12:33 PM

Tricky question. To get scale, what many people on here do is they switch from using D1 to using Durable Objects which allow you to spit the work and data storage up across many instances. However, you said "highly relational data" and that's more complex when your relationships are on the other side of a network hop. There are strategies for doing that like the Actors model (and Cloudflare has an Actor base class for Durable Object now). Even so, I would only recommend that if there is a relatively clean way to split up the work/storage (per user, per tenant, etc.). So for consistently "highly relational data", your best choice might be to use an external scale-up style database solution. Cloudflare uses Postgres for some of its own internal needs so there is good support for doing that. Check out Cloudflare's Hyperdrive offering if you think that's the way to go for you: https://developers.cloudflare.com/hyperdrive/

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

LLarry Tricky question. To get scale, what many people on here do is they switch from u...

Devoto•8/25/25, 2:37 PM

Thank you very much, this makes a lot of sense, I was actually looking at the exact same page you linked before seeing your message, and indeed it perfectly fits our needs as we have a postgres database already and it is almost impossible to break up the data model to an actor model since data model is tightly coupled. However, I have two questions:

1. Will cloudflare support higher DO or D1 in a pay per usage model similar to something like R2 in the future?
2. Does hyperdrive cache the transaction queries (select statements in a transaction) or is it just caching select statements outside transactions?

Many thanks!

DDevoto Thank you very much, this makes a lot of sense, I was actually looking at the ex...

Larry•8/25/25, 5:49 PM

1. I don’t know of any plans for that but I don’t work for Cloudflare and I never say never
2. My knowledge of hyperdrive consists of being aware that it exits and skimming that page a few times in the past. Sorry. Others on here may have hands on experience.

DDevoto Thank you very much, this makes a lot of sense, I was actually looking at the ex...

knickish•8/25/25, 5:54 PM

Only select statements outside of transactions are cached in Hyperdrive.

LLarry 1. I don’t know of any plans for that but I don’t work for Cloudflare and I neve...

Devoto•8/25/25, 7:26 PM

Excellent, thank you very much.

Kknickish Only select statements outside of transactions are cached in Hyperdrive.

Devoto•8/25/25, 7:26 PM

Appreciated!

Isaac McFadyen•8/25/25, 7:27 PM

Note that D1's existing limits are currently hard architectural limits (the 10GB limit I mean), it's not an artificial limit.

Sathyamshu•8/26/25, 7:02 AM

How to plan the database split for an application with more writes and reads? (cause the hard limit for a db is just 10 GB)
1. create databases with limited number of releatable tables
2. replicate the whole db structure, scale them horizontally and query them using the workers to map the right db
3. Please answer

which is the best practice? what does cloudflare recommend?
what's the next best alternative for this ?

SSathyamshu How to plan the database split for an application with more writes and reads? (c...

lambrospetrou•8/26/25, 8:04 AM

If you need a handful of DBs, D1 works fine, pre-create all the DBs upfront and attach them to your worker and use them as you wish.

If you need hundreds/thousands/millionds of these DBs, then it's much better to use SQLite Durable Objects. It's the same underlying storage and SQLite engine, but you have a much easier dev experience to use as many DBs as you wish.

Mech•8/26/25, 5:55 PM

Hey, is there a way to increase the 10 databases limit?

Llambrospetrou If you need a handful of DBs, D1 works fine, pre-create all the DBs upfront and ...

Mech•8/26/25, 5:56 PM

Oh, I see.

MMech Hey, is there a way to increase the 10 databases limit?

Chaika•8/26/25, 5:57 PM

Upgrade to Workers paid for 50k databases limit and you can ask for more after

Mech•8/26/25, 5:58 PM

Aha

Mech•8/26/25, 5:58 PM

I couldn't find in the docs

Chaika•8/26/25, 5:59 PM

ah all the limits are here: https://developers.cloudflare.com/d1/platform/limits/
Databases is top line

Cloudflare Docs

Limits

Cloudflare also offers other storage solutions such as Workers KV, Durable Objects, and R2. Each product has different advantages and limits. Refer to Choose a data or storage product to review which storage option is right for your use case.

Mech•8/26/25, 5:59 PM

Thank you

Silvan•8/27/25, 5:43 PM

So about the topic of d1 being able to randomly fail a query once in a blue moon. what is the exact error that it spits out? is it just a D1_ERRORD1_ERROR ?

Silvan•8/27/25, 5:46 PM

https://developers.cloudflare.com/d1/observability/debug-d1/#error-list
well i guess this good enough

Cloudflare Docs

Debug D1

D1 allows you to capture exceptions and log errors returned when querying a database. To debug D1, you will use the same tools available when debugging Workers.

tminima•8/27/25, 8:08 PM

Hey, guys, how can i pass a null value of a column in a python worker? Passing it as None doesn't work.

Ttminima Hey, guys, how can i pass a null value of a column in a python worker? Passing i...

lambrospetrou•8/28/25, 9:47 AM

Can you give an example of what you do at the moment and what is the error/problem? I don't know much about Python workers but seeing what you are doing will help us troubleshoot.

Llambrospetrou Can you give an example of what you do at the moment and what is the error/probl...

tminima•8/28/25, 9:49 AM

Hey, details are here: https://discord.com/channels/595317990191398933/1224715939581530112/1410355179383029800. Found a workaround.

Ttminima Hey, details are here: https://discord.com/channels/595317990191398933/122471593...

lambrospetrou•8/28/25, 9:52 AM

OK, cool, glad you got it sorted.

esdee•8/29/25, 5:34 PM

I have a repo that uses D1 with a very standard setup - D1 locally for developing, remote D1 for production. Does anyone know of an example repo that demonstrates good tests practices for D1. Test isolation, etc?

Tim Reynolds•8/30/25, 8:46 PM

I'm sure you get this feedback a lot but with workers now supporting preview deploys it would be good to have proper branch based D1 environments that we're linked to a preview build, today as far as I'm aware you can only have a single preview D1 instance used across all branches. This is quite limiting compared to using other setups like Neon or supabase with say vercel.

Eesdee I have a repo that uses D1 with a very standard setup - D1 locally for developin...

mw10013•8/30/25, 9:50 PM

I found this example to be helpful: https://github.com/cloudflare/workers-sdk/tree/4492eb0490588df736c25272ed2b279736462c9a/fixtures/vitest-pool-workers-examples/d1

GitHub

workers-sdk/fixtures/vitest-pool-workers-examples/d1 at 4492eb04905...

Home to Wrangler, the CLI for Cloudflare Workers® - cloudflare/workers-sdk

workers-sdk/fixtures/vitest-pool-workers-examples/d1 at 4492eb04905...

TTim Reynolds I'm sure you get this feedback a lot but with workers now supporting preview dep...

lambrospetrou•8/31/25, 7:00 AM

Yeah, this would be cool. We discussed about it a few times and once we implement fork/clone this will come next.
No defined ETA unfortunately though as this work did not yet start.

Llambrospetrou Yeah, this would be cool. We discussed about it a few times and once we implemen...

Tim Reynolds•8/31/25, 7:02 AM

Thanks for the insight, did seem like it would need a bunch of additional features which might stem from having timetravel which I think was added more recently

Nooc•8/31/25, 7:52 AM

Hello, I want to know if I can transfer D1 databases from one account to another account?

James•8/31/25, 11:30 AM

Not directly. The best way would be to export it and then re-import it on another account I imagine.

sirhype•8/31/25, 7:56 PM

Is there somewhere I can find all the supported D1 column types? I thought I knew what they were, but apparently TIMESTAMPTIMESTAMP is supported?

Ssirhype Is there somewhere I can find all the supported D1 column types? I thought I kne...

Chaika•8/31/25, 7:59 PM

It's all just Sqlite: https://www.sqlite.org/datatype3.html
There's no direct date/timestamp type, but it supports converting between text/real/int when you use date functions
Any type you specify that does not exist just fallbacks to numeric

sirhype•8/31/25, 7:59 PM

Ah, that's an interesting functionality. Thank you!

sirhype•8/31/25, 8:00 PM

Was confused because I saw this

sirhype•8/31/25, 8:00 PM

and it seemed to be treated as an integer

sirhype•8/31/25, 8:03 PM

but it's definitely not, so that makes sense now

Ssirhype and it seemed to be treated as an integer

Chaika•8/31/25, 8:06 PM

ah yea opps the default underlying type is numeric, but it can accept text/anything else so I always forget that

A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT. For the purposes of this paragraph, hexadecimal integer literals are not considered well-formed and are stored as TEXT. (This is done for historical compatibility with versions of SQLite prior to version 3.8.6 2014-08-15 where hexadecimal integer literals were first introduced into SQLite.) If a floating point value that can be represented exactly as an integer is inserted into a column with NUMERIC affinity, the value is converted into an integer. No attempt is made to convert NULL or BLOB values.

a bit messy. There's implicit conversion everywhere so you can get away with a lot

sirhype•8/31/25, 8:28 PM

Thanks for the explanation!

sirhype•8/31/25, 8:28 PM

Another question, what are "Default SQLite extensions"

sirhype•8/31/25, 8:29 PM

Does it include these?

Ssirhype Another question, what are "Default SQLite extensions"

Chaika•8/31/25, 8:44 PM

hmm yea that's questionable documentation

The Sqlite compilation for D1 is part of the open source workerd but hard to read: https://github.com/cloudflare/workerd/blob/4c42a4a9d3390c88e9bd977091c9d3395a6cd665/build/BUILD.sqlite3#L13

More simply, there's an specific allowlist of functions you're allowed to use with D1:
https://github.com/cloudflare/workerd/blob/4c42a4a9d3390c88e9bd977091c9d3395a6cd665/src/workerd/util/sqlite.c%2B%2B#L269
Json and fts5 are the only ones allowed for extensions/not base (and I've used both before, they def work).
They also only allow specific Virtual Tables for fts5 and fts5vocab: https://github.com/cloudflare/workerd/blob/4c42a4a9d3390c88e9bd977091c9d3395a6cd665/src/workerd/util/sqlite.c%2B%2B#L1155 so anything that depends on Virtual Tables (like zipfiles, csv, etc) will def not work

CChaika hmm yea that's questionable documentation The Sqlite compilation for D1 is part...

lambrospetrou•9/1/25, 4:31 PM

We made the docs a bit more explicit now, although it's a bit hard to list every single extension that is enabled by default in SQLite 3.47: https://developers.cloudflare.com/d1/sql-api/sql-statements/#supported-sqlite-extensions

Thanks to @Jun for the prompt updates!

sirhype•9/1/25, 8:32 PM

Thank you both so much! This was super helpful!

Hi! On a internal computer of our company, we would like to export our production D1 database every

Similar Threads

Similar Threads

Similar Threads