Hey there, just wondering, for highly relational data, how does 10GB size limit make sense? How do p

Hey there, just wondering, for highly relational data, how does 10GB size limit make sense? How do production apps go around that?

DDevoto Hey there, just wondering, for highly relational data, how does 10GB size limit ...

Larry•8/25/25, 12:33 PM

Tricky question. To get scale, what many people on here do is they switch from using D1 to using Durable Objects which allow you to spit the work and data storage up across many instances. However, you said "highly relational data" and that's more complex when your relationships are on the other side of a network hop. There are strategies for doing that like the Actors model (and Cloudflare has an Actor base class for Durable Object now). Even so, I would only recommend that if there is a relatively clean way to split up the work/storage (per user, per tenant, etc.). So for consistently "highly relational data", your best choice might be to use an external scale-up style database solution. Cloudflare uses Postgres for some of its own internal needs so there is good support for doing that. Check out Cloudflare's Hyperdrive offering if you think that's the way to go for you: https://developers.cloudflare.com/hyperdrive/

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

LLarry Tricky question. To get scale, what many people on here do is they switch from u...

DevotoOP•8/25/25, 2:37 PM

Thank you very much, this makes a lot of sense, I was actually looking at the exact same page you linked before seeing your message, and indeed it perfectly fits our needs as we have a postgres database already and it is almost impossible to break up the data model to an actor model since data model is tightly coupled. However, I have two questions:

1. Will cloudflare support higher DO or D1 in a pay per usage model similar to something like R2 in the future?
2. Does hyperdrive cache the transaction queries (select statements in a transaction) or is it just caching select statements outside transactions?

Many thanks!

DDevoto Thank you very much, this makes a lot of sense, I was actually looking at the ex...

Larry•8/25/25, 5:49 PM

1. I don’t know of any plans for that but I don’t work for Cloudflare and I never say never
2. My knowledge of hyperdrive consists of being aware that it exits and skimming that page a few times in the past. Sorry. Others on here may have hands on experience.

DDevoto Thank you very much, this makes a lot of sense, I was actually looking at the ex...

knickish•8/25/25, 5:54 PM

Only select statements outside of transactions are cached in Hyperdrive.

LLarry 1. I don’t know of any plans for that but I don’t work for Cloudflare and I neve...

DevotoOP•8/25/25, 7:26 PM

Excellent, thank you very much.

Kknickish Only select statements outside of transactions are cached in Hyperdrive.

DevotoOP•8/25/25, 7:26 PM

Appreciated!

Isaac McFadyen•8/25/25, 7:27 PM

Note that D1's existing limits are currently hard architectural limits (the 10GB limit I mean), it's not an artificial limit.

Sathyamshu•8/26/25, 7:02 AM

How to plan the database split for an application with more writes and reads? (cause the hard limit for a db is just 10 GB)
1. create databases with limited number of releatable tables
2. replicate the whole db structure, scale them horizontally and query them using the workers to map the right db
3. Please answer

which is the best practice? what does cloudflare recommend?
what's the next best alternative for this ?

SSathyamshu How to plan the database split for an application with more writes and reads? (c...

lambrospetrou•8/26/25, 8:04 AM

If you need a handful of DBs, D1 works fine, pre-create all the DBs upfront and attach them to your worker and use them as you wish.

If you need hundreds/thousands/millionds of these DBs, then it's much better to use SQLite Durable Objects. It's the same underlying storage and SQLite engine, but you have a much easier dev experience to use as many DBs as you wish.

Mech•8/26/25, 5:55 PM

Hey, is there a way to increase the 10 databases limit?

Llambrospetrou If you need a handful of DBs, D1 works fine, pre-create all the DBs upfront and ...

Mech•8/26/25, 5:56 PM

Oh, I see.

MMech Hey, is there a way to increase the 10 databases limit?

Chaika•8/26/25, 5:57 PM

Upgrade to Workers paid for 50k databases limit and you can ask for more after

Mech•8/26/25, 5:58 PM

Aha

Mech•8/26/25, 5:58 PM

I couldn't find in the docs

Chaika•8/26/25, 5:59 PM

ah all the limits are here: https://developers.cloudflare.com/d1/platform/limits/
Databases is top line

Cloudflare Docs

Limits

Cloudflare also offers other storage solutions such as Workers KV, Durable Objects, and R2. Each product has different advantages and limits. Refer to Choose a data or storage product to review which storage option is right for your use case.

Mech•8/26/25, 5:59 PM

Thank you

Silvan•8/27/25, 5:43 PM

So about the topic of d1 being able to randomly fail a query once in a blue moon. what is the exact error that it spits out? is it just a D1_ERRORD1_ERROR ?

Silvan•8/27/25, 5:46 PM

https://developers.cloudflare.com/d1/observability/debug-d1/#error-list
well i guess this good enough

Cloudflare Docs

Debug D1

D1 allows you to capture exceptions and log errors returned when querying a database. To debug D1, you will use the same tools available when debugging Workers.

tminima•8/27/25, 8:08 PM

Hey, guys, how can i pass a null value of a column in a python worker? Passing it as None doesn't work.

Ttminima Hey, guys, how can i pass a null value of a column in a python worker? Passing i...

lambrospetrou•8/28/25, 9:47 AM

Can you give an example of what you do at the moment and what is the error/problem? I don't know much about Python workers but seeing what you are doing will help us troubleshoot.

Llambrospetrou Can you give an example of what you do at the moment and what is the error/probl...

tminima•8/28/25, 9:49 AM

Hey, details are here: https://discord.com/channels/595317990191398933/1224715939581530112/1410355179383029800. Found a workaround.

Ttminima Hey, details are here: https://discord.com/channels/595317990191398933/122471593...

lambrospetrou•8/28/25, 9:52 AM

OK, cool, glad you got it sorted.

esdee•8/29/25, 5:34 PM

I have a repo that uses D1 with a very standard setup - D1 locally for developing, remote D1 for production. Does anyone know of an example repo that demonstrates good tests practices for D1. Test isolation, etc?

Tim Reynolds•8/30/25, 8:46 PM

I'm sure you get this feedback a lot but with workers now supporting preview deploys it would be good to have proper branch based D1 environments that we're linked to a preview build, today as far as I'm aware you can only have a single preview D1 instance used across all branches. This is quite limiting compared to using other setups like Neon or supabase with say vercel.

Eesdee I have a repo that uses D1 with a very standard setup - D1 locally for developin...

mw10013•8/30/25, 9:50 PM

I found this example to be helpful: https://github.com/cloudflare/workers-sdk/tree/4492eb0490588df736c25272ed2b279736462c9a/fixtures/vitest-pool-workers-examples/d1

GitHub

workers-sdk/fixtures/vitest-pool-workers-examples/d1 at 4492eb04905...

Home to Wrangler, the CLI for Cloudflare Workers® - cloudflare/workers-sdk

workers-sdk/fixtures/vitest-pool-workers-examples/d1 at 4492eb04905...

TTim Reynolds I'm sure you get this feedback a lot but with workers now supporting preview dep...

lambrospetrou•8/31/25, 7:00 AM

Yeah, this would be cool. We discussed about it a few times and once we implement fork/clone this will come next.
No defined ETA unfortunately though as this work did not yet start.

Llambrospetrou Yeah, this would be cool. We discussed about it a few times and once we implemen...

Tim Reynolds•8/31/25, 7:02 AM

Thanks for the insight, did seem like it would need a bunch of additional features which might stem from having timetravel which I think was added more recently

Nooc•8/31/25, 7:52 AM

Hello, I want to know if I can transfer D1 databases from one account to another account?

James•8/31/25, 11:30 AM

Not directly. The best way would be to export it and then re-import it on another account I imagine.

sirhype•8/31/25, 7:56 PM

Is there somewhere I can find all the supported D1 column types? I thought I knew what they were, but apparently TIMESTAMPTIMESTAMP is supported?

Ssirhype Is there somewhere I can find all the supported D1 column types? I thought I kne...

Chaika•8/31/25, 7:59 PM

It's all just Sqlite: https://www.sqlite.org/datatype3.html
There's no direct date/timestamp type, but it supports converting between text/real/int when you use date functions
Any type you specify that does not exist just fallbacks to numeric

sirhype•8/31/25, 7:59 PM

Ah, that's an interesting functionality. Thank you!

sirhype•8/31/25, 8:00 PM

Was confused because I saw this

sirhype•8/31/25, 8:00 PM

and it seemed to be treated as an integer

sirhype•8/31/25, 8:03 PM

but it's definitely not, so that makes sense now

Ssirhype and it seemed to be treated as an integer

Chaika•8/31/25, 8:06 PM

ah yea opps the default underlying type is numeric, but it can accept text/anything else so I always forget that

A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT. For the purposes of this paragraph, hexadecimal integer literals are not considered well-formed and are stored as TEXT. (This is done for historical compatibility with versions of SQLite prior to version 3.8.6 2014-08-15 where hexadecimal integer literals were first introduced into SQLite.) If a floating point value that can be represented exactly as an integer is inserted into a column with NUMERIC affinity, the value is converted into an integer. No attempt is made to convert NULL or BLOB values.

a bit messy. There's implicit conversion everywhere so you can get away with a lot

sirhype•8/31/25, 8:28 PM

Thanks for the explanation!

sirhype•8/31/25, 8:28 PM

Another question, what are "Default SQLite extensions"

sirhype•8/31/25, 8:29 PM

Does it include these?

Ssirhype Another question, what are "Default SQLite extensions"

Chaika•8/31/25, 8:44 PM

hmm yea that's questionable documentation

The Sqlite compilation for D1 is part of the open source workerd but hard to read: https://github.com/cloudflare/workerd/blob/4c42a4a9d3390c88e9bd977091c9d3395a6cd665/build/BUILD.sqlite3#L13

More simply, there's an specific allowlist of functions you're allowed to use with D1:
https://github.com/cloudflare/workerd/blob/4c42a4a9d3390c88e9bd977091c9d3395a6cd665/src/workerd/util/sqlite.c%2B%2B#L269
Json and fts5 are the only ones allowed for extensions/not base (and I've used both before, they def work).
They also only allow specific Virtual Tables for fts5 and fts5vocab: https://github.com/cloudflare/workerd/blob/4c42a4a9d3390c88e9bd977091c9d3395a6cd665/src/workerd/util/sqlite.c%2B%2B#L1155 so anything that depends on Virtual Tables (like zipfiles, csv, etc) will def not work

CChaika hmm yea that's questionable documentation The Sqlite compilation for D1 is part...

lambrospetrou•9/1/25, 4:31 PM

We made the docs a bit more explicit now, although it's a bit hard to list every single extension that is enabled by default in SQLite 3.47: https://developers.cloudflare.com/d1/sql-api/sql-statements/#supported-sqlite-extensions

Thanks to @Jun for the prompt updates!

sirhype•9/1/25, 8:32 PM

Thank you both so much! This was super helpful!

bren.app•9/1/25, 11:24 PM

one bit of feedback I've noticed about the D1 Studio! I have a bunch of tables without a rowid (since they have TEXT UUID primary keys), and I've noticed that the studio loves to append rowid to queries, making almost all interactions fail. The most annoying here is individual updates.

bren.app•9/1/25, 11:24 PM

(I will say I moved a bunch of this data from KV to D1 and it's been a marked improvement for my usecase)

Bbren.app one bit of feedback I've noticed about the D1 Studio! I have a bunch of tables w...

Brayden•9/1/25, 11:31 PM

Thanks for the feedback @bren.app, we'll take a look!

BBrayden Thanks for the feedback @bren.app, we'll take a look!

bren.app•9/1/25, 11:32 PM

Schema of the table

CREATE TABLE users(
    key TEXT UNIQUE PRIMARY KEY NOT NULL, -- Long Hex String (182 chars)
    name TEXT NOT NULL,
    role TEXT NOT NULL DEFAULT 'none',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) WITHOUT ROWID

CREATE TABLE users(
    key TEXT UNIQUE PRIMARY KEY NOT NULL, -- Long Hex String (182 chars)
    name TEXT NOT NULL,
    role TEXT NOT NULL DEFAULT 'none',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) WITHOUT ROWID

and the generated query when I modified a cell in the table

UPDATE "main"."users" SET "role" = 'none' WHERE "key" = '<ID>' RETURNING rowid, *

UPDATE "main"."users" SET "role" = 'none' WHERE "key" = '<ID>' RETURNING rowid, *

Bbren.app Schema of the table ``` CREATE TABLE users( key TEXT UNIQUE PRIMARY KEY NOT...

Visal .In•9/2/25, 2:18 AM

I will look into it.

Zack•9/3/25, 9:58 AM

Hey, so thought I would just raise this here because I have no idea what is happening. We have a dev D1 database that we deployed on a branch via terraform to test out some stuff. Our main branch ran some other terraform 2 weeks ago and deleted this dev D1 database, however a dev worker that was connected to this D1 database can still read this fine even today. I didn't even realise anything was wrong until I went to run new terraform today on this branch and it complained that the database was missing and needed to be re-created. Not sure what's going on here but feels weird (even though maybe nice for some people if they make a mistake) for the database to just sit around for 2 weeks when it shows as deleted in the dashboard. Who knows how long it''ll actually live on for. The database ID is/was 04a58760-3751-4337-bb3f-29a671a98cdd04a58760-3751-4337-bb3f-29a671a98cdd and the worker still connected to it is tiles-devtiles-dev in our account, specifically deployment 76803c3776803c37.

ZZack Hey, so thought I would just raise this here because I have no idea what is happ...

lambrospetrou•9/3/25, 12:45 PM

Your database is indeed marked deleted in our database (API) (on 2025-08-19 11:51:40.308558+00). When at that point of the delete request we cannot communicate with the DB itself, we have an async flow for the DB (DO) itself to mark itself as deleted on the next heartbeat. I am looking into why that part failed.

Dean•9/4/25, 2:59 AM

is DB performance still impacted in WNAM? seeing pretty slow queries although the query latency metrics looks normal.

Hey there, just wondering, for highly relational data, how does 10GB size limit make sense? How do p

Similar Threads

Similar Threads

Similar Threads