If you need to shard across tens thousands (or more) databases, please use SQLite DOs. Even if D1 sh

If you need to shard across tens thousands (or more) databases, please use SQLite DOs. Even if D1 ships dynamic bindings, it's not happening in just a few weeks. Use DOs that provide that scalability already.
The underlying SQLite features are identical.

Llambrospetrou If you need to shard across tens thousands (or more) databases, please use SQLit...

Silvan•4/3/25, 10:58 AM

is there a good example of this? at the end of the day you still need to bind the DO or am i forgetting something? running wrangler d1 create dosent seem like the issue to me personally

SSilvan is there a good example of this? at the end of the day you still need to bind th...

lambrospetrouOP•4/3/25, 11:21 AM

You only bind the DO namespace once, and create millions of Durable Object instances. With D1, you need to create each one individually and add it in your wrangler config and redeploy.

Silvan•4/3/25, 11:31 AM

well yea with the downside of needing to run migrations on every Do invocation because you cant know if its a new do/db right? at least thats what i did with my DO better-auth example

SSilvan well yea with the downside of needing to run migrations on every Do invocation b...

lambrospetrouOP•4/3/25, 11:45 AM

You do know if you check the storage and keep in memory while your DO is alive. Not sure why you would run your migrations all the time. All SQL schema tools/libraries record what migrations they already ran and don't do anything if there is nothing new.

Silvan•4/3/25, 11:53 AM

https://github.com/Sillyvan/better-auth-do-example/blob/main/src/server.ts

id still assume something like this brings a bit of overhead? but its not too big of a deal of course!

GitHub

better-auth-do-example/src/server.ts at main · Sillyvan/better-aut...

Contribute to Sillyvan/better-auth-do-example development by creating an account on GitHub.

Llambrospetrou If you need to shard across tens thousands (or more) databases, please use SQLit...

ac•4/3/25, 2:42 PM

I appreciate the response! Does this mean that for the tenant DBs we would have different DOs for each one, but we would only have to use one binding somehow?

ac•4/3/25, 2:42 PM

Forgive me if it's a dumb question, I haven't really used DOs but we have used D1 extensively

Aac I appreciate the response! Does this mean that for the tenant DBs we would have ...

Brayden•4/3/25, 3:42 PM

You could do something like the following for example where you have one Durable Object class in your wrangler binding (e.g. MY_DURABLE_OBJECT) but when you do the

idFromName

idFromName

you can pass something specific to the tenant you want. Each unique

id

id

creates its own isolated DO with its own SQLite storage.

let id = env.MY_DURABLE_OBJECT.idFromName('user_1');
let stub = env.MY_DURABLE_OBJECT.get(id);
return stub.fetch(request);

let id = env.MY_DURABLE_OBJECT.idFromName('user_1');
let stub = env.MY_DURABLE_OBJECT.get(id);
return stub.fetch(request);

So above I could change

user_1

user_1

to be any of my users and I'd be routed to their specific DO instance.

BBrayden You could do something like the following for example where you have one Durable...

ac•4/3/25, 4:03 PM

That's helpful, thank you!!

Aac That's helpful, thank you!!

lambrospetrouOP•4/3/25, 5:01 PM

Take a look into https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/ as well.

Cloudflare Docs

Control and data plane architectural pattern for Durable Objects ·...

Durable Objects are built on-top of Cloudflare Workers spanning several locations across our global infrastructure network.
Each Durable Object instance has its own durable storage persisted across requests, in-memory state, single-threaded execution, and can be placed in a specific region.

coinnp•4/4/25, 7:46 AM

Why I'm getting database disk image is malformed: SQLITE_CORRUPT [code: 7500]

Ccoinnp Why I'm getting database disk image is malformed: SQLITE_CORRUPT [code: 7500]

Acе•4/4/25, 10:58 AM

Because the database disk image is malformed

jakepez•4/4/25, 5:44 PM

Hi all - Does a D1 batch() count as a single (1) sub-request against the Workers limit or are the individual transactions in the batch counted against the subrequest limit?

Original message was deleted

jakepez•4/4/25, 5:57 PM

Thank you!

Victor•4/4/25, 9:50 PM

I'm weirdly getting WAF bot detection blocked on

api.cloudflare.com

api.cloudflare.com

hitting the d1

/query

/query

one of the rays:

92b3ef8f6aea1703-SJC

92b3ef8f6aea1703-SJC

Minions•4/4/25, 11:38 PM

Hi @Walshy

Minions•4/4/25, 11:38 PM

Nice to meet you

Minions•4/4/25, 11:38 PM

Having a nice weekend?

Minions•4/4/25, 11:55 PM

I need some advice.

Minions•4/4/25, 11:55 PM

Is it possible to access coudflare d1 database outside of worker/page?

MMinions Hi @Walshy

Cyb3r-Jak3•4/4/25, 11:58 PM

?pings

SuperHelpflare•4/4/25, 11:58 PM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

MMinions Is it possible to access coudflare d1 database outside of worker/page?

Cyb3r-Jak3•4/4/25, 11:59 PM

The general recommendation is to deploy a worker which acts as a HTTP API for your D1 database. You can just use the barebones Cloudflare HTTP API but that is slower

SSuperHelpflare Please ***do not ping*** community members for non-moderation reasons. Doing so ...

Minions•4/5/25, 12:01 AM

okay

CCyb3r-Jak3 The general recommendation is to deploy a worker which acts as a HTTP API for yo...

Minions•4/5/25, 12:07 AM

So, I need to access Worker, not D1 database, right?

Minions•4/5/25, 12:11 AM

How can I get the url of Cloudflare D1 database, so that I can set the db attribute of Payload CMS backend?

Cyb3r-Jak3•4/5/25, 12:12 AM

?d1-remote-connections

SuperHelpflare•4/5/25, 12:12 AM

Accessing D1 from outside workers:

D1 is based on SQLite, which does not natively support remote connections and isn't MySQL/PostgreSQL wire compatible, nor does D1 itself offer that functionality. As such, your options are:

1 -> The Cloudflare API Query Endpoint (https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database), with the upside of needing less setup, all requests have to go back to Cloudflare's core, so request latency is far from optimal from some locations. You would also be subject to the API Rate limit of 1200 requests per 5 minutes.

2 -> Create a worker to proxy requests to your D1 Database. Either tightly bound to your application/schema or loose and just accepting any query. This would allow your worker to execute potentially closer to your D1 Database and avoid the api rate limit.

CCyb3r-Jak3 ?d1-remote-connections

Minions•4/5/25, 12:12 AM

Oh, you're smart.

CCyb3r-Jak3 ?d1-remote-connections

Minions•4/5/25, 12:16 AM

I think option 2 is good,.

Minions•4/5/25, 12:17 AM

I tried with 'npx create-payload-app' project.

Minions•4/5/25, 12:18 AM

But the problem here is to implement admin dashboard

Disciple of Caoximeihao•4/5/25, 3:07 AM

Error: D1_ERROR: Currently processing a long-running export.

I need to synchronize the database from remote to local, but when I export, the system will be interrupted, my system needs to operate 24/7

DDisciple of Caoximeihao Error: D1_ERROR: Currently processing a long-running export. I need to synchron...

lambrospetrouOP•4/5/25, 7:20 AM

With the current design imports/exports block your database. Improvements of this are in the works but for now that's the state of it.

gruntlord6•4/6/25, 12:06 AM

Maybe use another database? Copy the data to it and use it for the export so it doesn’t block your application

Original message was deleted

gruntlord6•4/6/25, 12:32 AM

This looks really handy as a stop gap until dynamic bindings are eventually (hopefully) added

will smith•4/6/25, 3:48 PM

I see a bunch of questions in the past asking about per-user/tenant D1. How can I stay up to date on the topic over the next few months/years?

I've migrated off D1 for now but would like to migrate back once that's possible

James•4/6/25, 4:00 PM

There'll be a blog post I imagine when it's actually possible. For now, if you want to do that, you can with Durable Objects, but it's basically not possible in any ergonomic way with D1. Nor would I recommend it for the vast majority of apps over a more traditional DB solution.

martinklepsch•4/7/25, 11:18 AM

Hey all, I saw @yusukebe's recent Hono Vite template and wrote a prompt to add D1 (with drizzle), the prompt worked really well for me — would love to hear if it works for others too!

https://ctxs.ai/weekly/d1-database-setup-for-classifieds-app-a0m502

Deltachaos•4/7/25, 12:02 PM

Regarding scaleing: So if D1 uses SQLite under the hood, does it scale for databases with hundrets of millions of records within one table?

DDeltachaos Regarding scaleing: So if D1 uses SQLite under the hood, does it scale for datab...

lambrospetrouOP•4/7/25, 12:10 PM

"Scaling" doesn't mean much without specifying for which queries. If you are queries use proper indices, and provide

WHERE

WHERE

filters to not scan the table for every query and limit the results you get back, it can do fine. We have folks with millions of rows.
The caveat is that if at any point you try to modify millions of rows, things will not work well.

So, having millions of rows is not an issue (assuming they fit the storage limits), but modifying in your query millions of rows each time will lead to issues most likely.

Zack•4/7/25, 2:17 PM

So I have a question around D1 wrt dropping large tables. We have tables that sum to ~3.5 million rows and ~2.5GB each. Generally we will only have one of these, but when we need to update the data stored in them (they are used as a lookup of sorts) we will create a brand new table, switch to that, then drop the old table. The last part of this is causing us some significant issues and I am not sure how to work around it. While the

DROP TABLE {name}

DROP TABLE {name}

query is running, the table becomes completely unresponsive and can't serve other queries. We always get returned a failure for this query, whether using worker bindings or the API, however sometimes it actually does work and sometimes it doesn't (presumably the operation times out and a rollback is initiated). I wanted to know if this is something you guys are aware of, and if there are plans to improve this, because right now we can't really delete our old data without causing a temporary outage, and we may have to even try multiple times to delete the table. Cheers!

Corsix👾•4/7/25, 2:32 PM

It says in the documentation that the response from a .batch() call should be a array but I'm getting a json object with numberd keys for the batch statments? Is this something new?

Corsix👾•4/7/25, 2:33 PM

rn_image_picker_lib_temp_3a29382b-9326-42a9-8e53-22a8161ec4d8.jpg

ZZack So I have a question around D1 wrt dropping large tables. We have tables that su...

lambrospetrouOP•4/7/25, 2:40 PM

Yes, we are aware, and unfortunately it's not something there is an obvious solution right now. When you delete 3.5 million of rows, that's causing pretty much writes for all those SQLite pages, and at some point you are hitting the Durable Object CPU limits (a D1 database is backed by a DO). Yes, each query is in a transaction and if it fails it rolls back.

The only recommendation we have for now is that those deletions/writes should be done in lower chunks.

Llambrospetrou Yes, we are aware, and unfortunately it's not something there is an obvious solu...

Zack•4/7/25, 2:41 PM

Right, and that's something we considered. However that gets pretty expensive, given that a

DROP TABLE

DROP TABLE

is 1 write, and

DELETE FROM

DELETE FROM

is 1 write per row deleted :/

CCorsix👾Click to see attachment

lambrospetrouOP•4/7/25, 2:41 PM

How are you printing that? An array in JS is an object with numbers as keys, and those numbers are the array's indices.

Zack•4/7/25, 2:42 PM

Maybe something could be done around that to make it less prohibitively expensive to delete tables in chunks, while this problem exists

Mmarshallswain Just going off the numbers in Zack's message (2 after yours) He reports "~3.5 mi...

lambrospetrouOP•4/7/25, 2:44 PM

A row value can be 8 bytes, or 2MBs. Really it depends.

If you need to shard across tens thousands (or more) databases, please use SQLite DOs. Even if D1 sh

Similar Threads

Similar Threads

Similar Threads