have a worker dump the tables

Samantas5855•8/19/25, 8:44 PM

Is the Maximum SQL statement length not a thing anymore? I was able to write over 200kb in a single cell with a single bind statement

SSamantas5855 Is the Maximum SQL statement length not a thing anymore? I was able to write ove...

lambrospetrou•8/19/25, 8:56 PM

Each row fits 2MB. Did you have the full 200KB hardcoded in the query SQL statement or you said you used the bind()bind()?
What was your query?

Llambrospetrou Each row fits 2MB. Did you have the full 200KB hardcoded in the query SQL statem...

Samantas5855•8/19/25, 9:13 PM

Does the query with variables count the questionmarks only for length? I did use variables and put a bunch of data in bind()

SSamantas5855 Does the query with variables count the questionmarks only for length? I did use...

lambrospetrou•8/19/25, 10:20 PM

Yes, the limit applies to the statement, not the data bound.

Jun•8/20/25, 5:00 PM

FYI: We've updated the D1 error docs today!

https://developers.cloudflare.com/d1/observability/debug-d1/#error-list

Ccloudkite We are building a db per customer architecture and were holding out for dynamic ...

flof.fly•8/20/25, 10:02 PM

Tom•8/21/25, 8:54 AM

Hey, yesterday, during our burst load, we received 153 "D1_ERROR: D1 DB is overloaded. Too many requests queued." on our d1 db while serving about 161k queries for ~30 minutes. It's a lot of queries and it seems like we managed to just stand of the edge of a single D1 db capacity. Do you guys at Cloudflare see more statistics about databases hidden to us users that ought to be interesting?

Tom•8/21/25, 8:54 AM

The ID is 94f2eff2-6fa1-4350-bb8f-121c5da3ed3e

GuillaumeLakano•8/22/25, 3:59 PM

Hi! On a internal computer of our company, we would like to export our production D1 database every 10 minutes ( from the CLI wrangler d1 export ). Is someone from the CF staff can confirm us it's tolerated and this will not be blocked or against the CF policies please?

GGuillaumeLakano Hi! On a internal computer of our company, we would like to export our productio...

lambrospetrou•8/22/25, 4:06 PM

There is no policy against doing that, but keep in mind that whle exporting your database is blocked from serving any queries, so doing it every 10 minutes, especially if it's more than a few hundred MBs, it's not going to be a nice experience to your users.

Original message was deleted

lambrospetrou•8/22/25, 5:31 PM

Not at the moment. Hopefully we will get to refactoring the whole export flow soon, and that won't be necessary.

Larry•8/22/25, 5:31 PM

With read-replicas, it should be a lot easier. Detach a replica and export that.

GGuillaumeLakano Hi! On a internal computer of our company, we would like to export our productio...

gruntlord6OP•8/22/25, 10:32 PM

Every 10 minutes seems excessive since you can just read the database directly or use time travel for restores between your backup periods. Is there something in the use case that requires that frequency?

Ggruntlord6 have a worker dump the tables

Miku•8/23/25, 11:13 AM

use worker to export?

MMiku use worker to export?

gruntlord6OP•8/23/25, 1:20 PM

workers can read table entries so you can just have it read the tables and save it to r2

Advany•8/25/25, 9:08 AM

how can I protect d1 that it can't be access using wrangler? (I want to prevent claude code from deleting data or dropping tables etc)

Devoto•8/25/25, 10:45 AM

Hey there, just wondering, for highly relational data, how does 10GB size limit make sense? How do production apps go around that?

DDevoto Hey there, just wondering, for highly relational data, how does 10GB size limit ...

Larry•8/25/25, 12:33 PM

Tricky question. To get scale, what many people on here do is they switch from using D1 to using Durable Objects which allow you to spit the work and data storage up across many instances. However, you said "highly relational data" and that's more complex when your relationships are on the other side of a network hop. There are strategies for doing that like the Actors model (and Cloudflare has an Actor base class for Durable Object now). Even so, I would only recommend that if there is a relatively clean way to split up the work/storage (per user, per tenant, etc.). So for consistently "highly relational data", your best choice might be to use an external scale-up style database solution. Cloudflare uses Postgres for some of its own internal needs so there is good support for doing that. Check out Cloudflare's Hyperdrive offering if you think that's the way to go for you: https://developers.cloudflare.com/hyperdrive/

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

LLarry Tricky question. To get scale, what many people on here do is they switch from u...

Devoto•8/25/25, 2:37 PM

Thank you very much, this makes a lot of sense, I was actually looking at the exact same page you linked before seeing your message, and indeed it perfectly fits our needs as we have a postgres database already and it is almost impossible to break up the data model to an actor model since data model is tightly coupled. However, I have two questions:

1. Will cloudflare support higher DO or D1 in a pay per usage model similar to something like R2 in the future?
2. Does hyperdrive cache the transaction queries (select statements in a transaction) or is it just caching select statements outside transactions?

Many thanks!

DDevoto Thank you very much, this makes a lot of sense, I was actually looking at the ex...

Larry•8/25/25, 5:49 PM

1. I don’t know of any plans for that but I don’t work for Cloudflare and I never say never
2. My knowledge of hyperdrive consists of being aware that it exits and skimming that page a few times in the past. Sorry. Others on here may have hands on experience.

DDevoto Thank you very much, this makes a lot of sense, I was actually looking at the ex...

knickish•8/25/25, 5:54 PM

Only select statements outside of transactions are cached in Hyperdrive.

LLarry 1. I don’t know of any plans for that but I don’t work for Cloudflare and I neve...

Devoto•8/25/25, 7:26 PM

Excellent, thank you very much.

Kknickish Only select statements outside of transactions are cached in Hyperdrive.

Devoto•8/25/25, 7:26 PM

Appreciated!

Isaac McFadyen•8/25/25, 7:27 PM

Note that D1's existing limits are currently hard architectural limits (the 10GB limit I mean), it's not an artificial limit.

Sathyamshu•8/26/25, 7:02 AM

How to plan the database split for an application with more writes and reads? (cause the hard limit for a db is just 10 GB)
1. create databases with limited number of releatable tables
2. replicate the whole db structure, scale them horizontally and query them using the workers to map the right db
3. Please answer

which is the best practice? what does cloudflare recommend?
what's the next best alternative for this ?

SSathyamshu How to plan the database split for an application with more writes and reads? (c...

lambrospetrou•8/26/25, 8:04 AM

If you need a handful of DBs, D1 works fine, pre-create all the DBs upfront and attach them to your worker and use them as you wish.

If you need hundreds/thousands/millionds of these DBs, then it's much better to use SQLite Durable Objects. It's the same underlying storage and SQLite engine, but you have a much easier dev experience to use as many DBs as you wish.

Mech•8/26/25, 5:55 PM

Hey, is there a way to increase the 10 databases limit?

Llambrospetrou If you need a handful of DBs, D1 works fine, pre-create all the DBs upfront and ...

Mech•8/26/25, 5:56 PM

Oh, I see.

MMech Hey, is there a way to increase the 10 databases limit?

Chaika•8/26/25, 5:57 PM

Upgrade to Workers paid for 50k databases limit and you can ask for more after

Mech•8/26/25, 5:58 PM

Aha

Mech•8/26/25, 5:58 PM

I couldn't find in the docs

Chaika•8/26/25, 5:59 PM

ah all the limits are here: https://developers.cloudflare.com/d1/platform/limits/
Databases is top line

Cloudflare Docs

Limits

Cloudflare also offers other storage solutions such as Workers KV, Durable Objects, and R2. Each product has different advantages and limits. Refer to Choose a data or storage product to review which storage option is right for your use case.

Mech•8/26/25, 5:59 PM

Thank you

Silvan•8/27/25, 5:43 PM

So about the topic of d1 being able to randomly fail a query once in a blue moon. what is the exact error that it spits out? is it just a D1_ERRORD1_ERROR ?

Silvan•8/27/25, 5:46 PM

https://developers.cloudflare.com/d1/observability/debug-d1/#error-list
well i guess this good enough

Cloudflare Docs

Debug D1

D1 allows you to capture exceptions and log errors returned when querying a database. To debug D1, you will use the same tools available when debugging Workers.

tminima•8/27/25, 8:08 PM

Hey, guys, how can i pass a null value of a column in a python worker? Passing it as None doesn't work.

Ttminima Hey, guys, how can i pass a null value of a column in a python worker? Passing i...

lambrospetrou•8/28/25, 9:47 AM

Can you give an example of what you do at the moment and what is the error/problem? I don't know much about Python workers but seeing what you are doing will help us troubleshoot.

Llambrospetrou Can you give an example of what you do at the moment and what is the error/probl...

tminima•8/28/25, 9:49 AM

Hey, details are here: https://discord.com/channels/595317990191398933/1224715939581530112/1410355179383029800. Found a workaround.

Ttminima Hey, details are here: https://discord.com/channels/595317990191398933/122471593...

lambrospetrou•8/28/25, 9:52 AM

OK, cool, glad you got it sorted.

esdee•8/29/25, 5:34 PM

I have a repo that uses D1 with a very standard setup - D1 locally for developing, remote D1 for production. Does anyone know of an example repo that demonstrates good tests practices for D1. Test isolation, etc?

Tim Reynolds•8/30/25, 8:46 PM

I'm sure you get this feedback a lot but with workers now supporting preview deploys it would be good to have proper branch based D1 environments that we're linked to a preview build, today as far as I'm aware you can only have a single preview D1 instance used across all branches. This is quite limiting compared to using other setups like Neon or supabase with say vercel.

Eesdee I have a repo that uses D1 with a very standard setup - D1 locally for developin...

mw10013•8/30/25, 9:50 PM

I found this example to be helpful: https://github.com/cloudflare/workers-sdk/tree/4492eb0490588df736c25272ed2b279736462c9a/fixtures/vitest-pool-workers-examples/d1

GitHub

workers-sdk/fixtures/vitest-pool-workers-examples/d1 at 4492eb04905...

Home to Wrangler, the CLI for Cloudflare Workers® - cloudflare/workers-sdk

workers-sdk/fixtures/vitest-pool-workers-examples/d1 at 4492eb04905...

TTim Reynolds I'm sure you get this feedback a lot but with workers now supporting preview dep...

lambrospetrou•8/31/25, 7:00 AM

Yeah, this would be cool. We discussed about it a few times and once we implement fork/clone this will come next.
No defined ETA unfortunately though as this work did not yet start.

Llambrospetrou Yeah, this would be cool. We discussed about it a few times and once we implemen...

Tim Reynolds•8/31/25, 7:02 AM

Thanks for the insight, did seem like it would need a bunch of additional features which might stem from having timetravel which I think was added more recently

Nooc•8/31/25, 7:52 AM

Hello, I want to know if I can transfer D1 databases from one account to another account?

James•8/31/25, 11:30 AM

Not directly. The best way would be to export it and then re-import it on another account I imagine.

sirhype•8/31/25, 7:56 PM

Is there somewhere I can find all the supported D1 column types? I thought I knew what they were, but apparently

TIMESTAMP

TIMESTAMP

is supported?

Ssirhype Is there somewhere I can find all the supported D1 column types? I thought I kne...

Chaika•8/31/25, 7:59 PM

It's all just Sqlite: https://www.sqlite.org/datatype3.html
There's no direct date/timestamp type, but it supports converting between text/real/int when you use date functions
Any type you specify that does not exist just fallbacks to numeric

sirhype•8/31/25, 7:59 PM

Ah, that's an interesting functionality. Thank you!

sirhype•8/31/25, 8:00 PM

Was confused because I saw this

have a worker dump the tables

Similar Threads

Similar Threads

Similar Threads