I just found out about the 10 GB limit on D1. And I think I'll need at least 30-50gb. I know D1's c

I just found out about the 10 GB limit on D1. And I think I'll need at least 30-50gb.

I know D1's can horizontally scale but I can't find any docs on how you would create a D1 instance per tentant/organization in a worker. Does anyone has any examples or docs they could point to?

CChris Jayden I just found out about the 10 GB limit on D1. And I think I'll need at least 30-...

lambrospetrou•1/14/25, 9:22 AM

At the moment you cannot create D1 databases dynamically within your worker code. You need to create one through the D1 Rest API or the dashboard, and then add it in your wrangler.toml (and/or dynamically patch your worker on the fly).

So, in your case you probably just need to create 5-10 dbs and add them in your wrangler.

There is no automatic migration of the data from your existing DB to those new dbs, just to be clear though.

Llambrospetrou At the moment you cannot create D1 databases dynamically within your worker code...

Chris JaydenOP•1/14/25, 9:27 AM

Thanks for your quick answer. I think I'll be switching from D1 to a database that has a higher limit, until Workers can dynamically handle new D1 instances for tenants.

Chris JaydenOP•1/14/25, 9:29 AM

~~I did see in the docs just now that you can increase a limit, but there's not much information regarding that; https://developers.cloudflare.com/d1/platform/limits/~~

Actually, scratch that, the docs say

Note that the 10 GB limit of a D1 database cannot be further increased.

CChris Jayden Thanks for your quick answer. I think I'll be switching from D1 to a database th...

lambrospetrou•1/14/25, 9:30 AM

BTW, Durable Objects are the same underlying SQL engine, and you can create as many of those as you need just within your Worker code, so if that works for you, it would probably be less work than migrating to a whole other system.

Llambrospetrou BTW, Durable Objects are the same underlying SQL engine, and you can create as m...

Chris JaydenOP•1/14/25, 9:34 AM

That's interesting. What are the differences?

Chris JaydenOP•1/14/25, 9:35 AM

I see the sqlite is quite a recent thing in DO? https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-storage/#sqlite-storage-backend

Chris JaydenOP•1/14/25, 9:36 AM

This is a helpful blog I just found that helps to explain that use case a bit more; https://www.lambrospetrou.com/articles/durable-objects-cloudflare/

Lambros Petrou personal website

Durable Objects (DO) — Unlimited single-threaded servers spread acr...

Understand Cloudflare Durable Objects and use them to simplify your architecture.

CChris Jayden I see the sqlite is quite a recent thing in DO? https://developers.cloudflare.co...

lambrospetrou•1/14/25, 9:40 AM

Yeah, SQLite in DO launched a few months ago, but it has been used by D1 for more than a year now.

If you also want to see how to think about using many DOs check out https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/

I have a real example implementing that pattern explained in https://www.lambrospetrou.com/articles/tiddlyflare/

Also https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-storage/#sql-in-durable-objects-vs-d1 has some light comparison between the two.

Chris JaydenOP•1/14/25, 9:40 AM

Haha i just noticed

Chris JaydenOP•1/14/25, 9:40 AM

that's your blog

Chris JaydenOP•1/14/25, 9:40 AM

what a weird coincidence

Chris JaydenOP•1/14/25, 9:41 AM

these articles are terrific @lambrospetrou

CChris Jayden I just found out about the 10 GB limit on D1. And I think I'll need at least 30-...

Devils•1/14/25, 11:20 AM

Personally my stack right now is 3 VPS’s all with around 80gb of storage running YugabyteDB’s..

Devils•1/14/25, 11:20 AM

But like lambrospetrou said, it would be much harder to switch or migrate to another database.

Devils•1/14/25, 11:21 AM

I’m having to build a loadbalancer and pooler.

Devils•1/14/25, 11:21 AM

Even managed databases are eh.

DDevils Personally my stack right now is 3 VPS’s all with around 80gb of storage running...

Chris JaydenOP•1/14/25, 1:21 PM

Appreciate you chiming in. Noted ! i'm really intrigued by Durable Objects. They seem underrated/underappreciated. I thought they were mostly for real-time stuff, but it seems like a great way to scale!

FFriday Candour great to scale? elaborate.

Isaac McFadyen•1/14/25, 6:03 PM

You can create unlimited Durable Objects each with their own SQLite database.

Isaac McFadyen•1/14/25, 6:08 PM

You're correct, unlimited is never unlimited, but in this case "unlimited" is so high that if you reached that point you'd have an Enterprise agreement with them and could negotiate higher limits anyway.
Using D1 via API is slow because of the architecture, it's not some grand scheme to force you into using a Worker. D1 is located on Cloudflare's edge network but the API routes through the cores (because it was designed well before D1 and that was how it was built).

Chaika•1/14/25, 6:11 PM

They run on the same proxy machine that handles your connection

Chaika•1/14/25, 6:11 PM

so yes, very edge, about as edge as you can get lol

Ashkan•1/14/25, 11:06 PM

Hi Dear CF Team, In the D1 metric tab I think it lacks some informative data, such as what are the top 5 or most read tables (in some situations when an cyber attack happens the most read tables are the clues for security location hole). (feedback)

AAshkan Hi Dear CF Team, In the D1 metric tab I think it lacks some informative data, su...

Chaika•1/14/25, 11:09 PM

D1 has the insights command in wrangler: https://developers.cloudflare.com/d1/observability/metrics-analytics/#query-insights, which can show the most executed queries, slowest, reading the most rows/writing the most rows, etc

Cloudflare Docs

Metrics and analytics · Cloudflare D1 docs

D1 exposes database analytics that allow you to inspect query volume, query latency, and storage size across all and/or each database in your account.

CChaika D1 has the insights command in wrangler: https://developers.cloudflare.com/d1/ob...

Ashkan•1/14/25, 11:16 PM

Thank you for your reply. it works for developers who are familiar with wrangler and so on.

Alex Patterson•1/14/25, 11:18 PM

I have setup a project using

pnpx wrangler d1 create cove

pnpx wrangler d1 create cove

It doesn't seem like I can use this with a

--local

--local

flag, so it doesn't seem like we can truly use this as local development only. As in NOTHING created remote.

Ashkan•1/14/25, 11:46 PM

@Alex Patterson , If I understood your problem well, in order to create or signal cloudflare to create your database remotely you must use

--remote

--remote

flag when executing your command for example

wrangler d1 execute  YOUR_DB_NAME --file ./database/script.sql --remote

wrangler d1 execute  YOUR_DB_NAME --file ./database/script.sql --remote

AAlex Patterson I have setup a project using `pnpx wrangler d1 create cove` It doesn't seem lik...

Max (@rozenmd)•1/15/25, 9:26 AM

there's no create step for local dbs, you should be able to just start querying

cosbgn•1/15/25, 10:06 AM

I cache my D1 connection like this (on workers):

let cached_db = null;
export const useDb = () => {
  if (!cached_db) {
    const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB
    cached_db = drizzle(binding, { schema })
  }
  return cached_db
}

let cached_db = null;
export const useDb = () => {
  if (!cached_db) {
    const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB
    cached_db = drizzle(binding, { schema })
  }
  return cached_db
}

Can this be the cause of:

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

cosbgn•1/15/25, 10:06 AM

Should I avoid caching the connection?

Ccosbgn I cache my D1 connection like this (on workers): ```js let cached_db = null; ex...

lambrospetrou•1/15/25, 10:08 AM

I don't know how Drizzle works internally, but D1 doesn't have the concept of a long-lived connection like a traditional database. Every query you do sends a request to the database using connections managed by the runtime.
Our recommendation is to always use the fresh

env.{DATABASE_BINDING}

env.{DATABASE_BINDING}

you get in each request, otherwise you will definitely hit issues at some point.

This is what their example is doing as well: https://orm.drizzle.team/docs/connect-cloudflare-d1#step-2---initialize-the-driver-and-make-a-query
Looking at https://github.com/drizzle-team/drizzle-orm/blob/main/drizzle-orm/src/d1/driver.ts#L39 it's not doing expensive stuff, so just create it per request.

Llambrospetrou I don't know how Drizzle works internally, but D1 doesn't have the concept of a ...

lambrospetrou•1/15/25, 10:10 AM

The specific error you got above is not relevant to that anyway, that error says that your query took too long and was killed.
Check if you are doing any kind of scans over thousands/millions of rows each time.

cosbgn•1/15/25, 10:14 AM

Does efficiency 0 means it's really bad?

 "avgRowsRead": 0,
    "totalRowsRead": 13401,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 0.33640454375886003,
    "totalDurationMs": 4508.830100000001,
    "numberOfTimesRun": 13403,
    "queryEfficiency": 0

 "avgRowsRead": 0,
    "totalRowsRead": 13401,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 0.33640454375886003,
    "totalDurationMs": 4508.830100000001,
    "numberOfTimesRun": 13403,
    "queryEfficiency": 0

Ccosbgn Does efficiency 0 means it's really bad? ``` "avgRowsRead": 0, "totalRowsR...

lambrospetrou•1/15/25, 10:16 AM

Your query took

0.3ms

0.3ms

on average so why are you worried about it?

cosbgn•1/15/25, 10:16 AM

mainly because I keep getting this:

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

cosbgn•1/15/25, 10:16 AM

like every day

Ccosbgn mainly because I keep getting this: `[request error] [unhandled] [500] D1_ERROR:...

lambrospetrou•1/15/25, 10:17 AM

Then check the query that's failing, what it's doing. Get insights to see your slowest queries: https://developers.cloudflare.com/d1/observability/metrics-analytics/#query-insights

cosbgn•1/15/25, 10:18 AM

the thing is that I get the error on different queries, it's not one specific

cosbgn•1/15/25, 10:18 AM

anyways I'll run the query insight till I find something. Thanks for the tips!

Robin Genz•1/15/25, 11:42 AM

Hey

I get the following error when I try to apply a migration:

✘ [ERROR] Migration 0043_cultured_the_stranger.sql failed with the following errors:
  
  
  
  ✘ [ERROR] <html>
  
  
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</... (length = 168)
    POST /accounts/***/d1/database/<DATABASE_ID>/query -> 504 Gateway Time-out

✘ [ERROR] Migration 0043_cultured_the_stranger.sql failed with the following errors:
  
  
  
  ✘ [ERROR] <html>
  
  
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</... (length = 168)
    POST /accounts/***/d1/database/<DATABASE_ID>/query -> 504 Gateway Time-out

I am using the latest version of Wrangler. The migration is very simple:

ALTER TABLE `app_devices` DROP COLUMN `force_app_bundle_id`;
ALTER TABLE `app_devices` DROP COLUMN `force_app_channel_id`;

ALTER TABLE `app_devices` DROP COLUMN `force_app_bundle_id`;
ALTER TABLE `app_devices` DROP COLUMN `force_app_channel_id`;

How can I fix this? It is currently blocking our production deployments.

MMax (@rozenmd)there's no create step for local dbs, you should be able to just start querying

Alex Patterson•1/15/25, 11:52 AM

I totally see now I have have been going through the tutorial and it seemed like the only way to start locally was with

npx wrangler d1 create

npx wrangler d1 create

. Without a wrangler project with the DB in it that is probably true, but once I have one it is no problem.

I have restructured my local project to all point to a shared local persistence area

--persist-to ../../../cloudflare/.workers/cache/

--persist-to ../../../cloudflare/.workers/cache/

which will allow for access to the DB from multiple microservices.

I now have a D1 only project file with just the required toml file for the db, prior to doing this it was generating a DB per running worker which didn't have the correct data because they were both in their own database. Is there a best spot to update the docs for this setup? Maybe at least a knowlege add to the local development area?

Alex Patterson•1/15/25, 11:53 AM

here is how I handle the shared persistence.

AAlex Patterson I totally see now I have have been going through the tutorial and it seemed like...

Moh•1/15/25, 1:13 PM

i came here to ask about this, if you have a d1 db shared between workers, where do the migrations go?

MMoh i came here to ask about this, if you have a d1 db shared between workers, where...

lambrospetrou•1/15/25, 1:24 PM

Your migrations should be owned by 1 process/application/worker. Choose which one and put them there. A database can be used by many apps/workers but one of them is the owner of the database. Same as any other application using a database.

Moh•1/15/25, 1:29 PM

i was thinking of workers as smaller than a service, but I guess I should treat them as microservices

Moh•1/15/25, 1:29 PM

in which case I should probably not share the DB

let cached_db = null; export const useDb = () => { if (!cached_db) { const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB cached_db = drizzle(binding, { schema }) } return cached_db }

"avgRowsRead": 0, "totalRowsRead": 13401, "avgRowsWritten": 0, "totalRowsWritten": 0, "avgDurationMs": 0.33640454375886003, "totalDurationMs": 4508.830100000001, "numberOfTimesRun": 13403, "queryEfficiency": 0

✘ [ERROR] Migration 0043_cultured_the_stranger.sql failed with the following errors: ✘ [ERROR] <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</... (length = 168) POST /accounts/***/d1/database/<DATABASE_ID>/query -> 504 Gateway Time-out

I just found out about the 10 GB limit on D1. And I think I'll need at least 30-50gb. I know D1's c

Similar Threads

Similar Threads

Similar Threads