Anyway to cluster multiple D1 Databases together, or would I essentially have to create custom middl

D

DevilsOP•1/13/25, 11:28 PM

Looking to scale, but I'd rather just migrate over to something like YugabyteDB or TiDB and host my own database infastructure if I'm going to have to make custom middleware.

DDevils Anyway to cluster multiple D1 Databases together, or would I essentially have to...

1

1984 Ford Laser•1/13/25, 11:31 PM

Most of Cloudflare's dev platform requires some sort of custom middleware at some point. Durable Objects might be a solution to your problem, but again you'll need some amount of custom middleware

11984 Ford Laser Most of Cloudflare's dev platform requires some sort of custom middleware at som...

D

DevilsOP•1/13/25, 11:32 PM

Interesting, great to know now than later.. didn't want to dig to deep into something that I'm just going to have to migrate away from anyways.

D

DevilsOP•1/13/25, 11:32 PM

Working on API layers so.

D

DevilsOP•1/13/25, 11:32 PM

D

DevilsOP•1/13/25, 11:35 PM

The only products from Cloudflare I've really implemented is R2, coming from Backblaze, the costs for R2 are a lot cheaper. I had no idea that most of the products actually needed workers and stuff to work out of the box.

1

1984 Ford Laser•1/13/25, 11:37 PM

The idea is that you can build the system to work exactly how you need it. That might not work for every use case, but it means you have so much freedom to build to suit your use case

11984 Ford Laser The idea is that you can build the system to work exactly how you need it. That ...

D

DevilsOP•1/13/25, 11:38 PM

I think its just the fact that not only are you paying for D1 costs, but workers as well. Nonetheless I can understand it. Thanks my friend.

C

Chris Jayden•1/14/25, 8:56 AM

I just found out about the 10 GB limit on D1. And I think I'll need at least 30-50gb.

I know D1's can horizontally scale but I can't find any docs on how you would create a D1 instance per tentant/organization in a worker. Does anyone has any examples or docs they could point to?

CChris Jayden I just found out about the 10 GB limit on D1. And I think I'll need at least 30-...

L

lambrospetrou•1/14/25, 9:22 AM

At the moment you cannot create D1 databases dynamically within your worker code. You need to create one through the D1 Rest API or the dashboard, and then add it in your wrangler.toml (and/or dynamically patch your worker on the fly).

So, in your case you probably just need to create 5-10 dbs and add them in your wrangler.

There is no automatic migration of the data from your existing DB to those new dbs, just to be clear though.

Llambrospetrou At the moment you cannot create D1 databases dynamically within your worker code...

C

Chris Jayden•1/14/25, 9:27 AM

Thanks for your quick answer. I think I'll be switching from D1 to a database that has a higher limit, until Workers can dynamically handle new D1 instances for tenants.

C

Chris Jayden•1/14/25, 9:29 AM

~~I did see in the docs just now that you can increase a limit, but there's not much information regarding that; https://developers.cloudflare.com/d1/platform/limits/~~

Actually, scratch that, the docs say

Note that the 10 GB limit of a D1 database cannot be further increased.

CChris Jayden Thanks for your quick answer. I think I'll be switching from D1 to a database th...

L

lambrospetrou•1/14/25, 9:30 AM

BTW, Durable Objects are the same underlying SQL engine, and you can create as many of those as you need just within your Worker code, so if that works for you, it would probably be less work than migrating to a whole other system.

Llambrospetrou BTW, Durable Objects are the same underlying SQL engine, and you can create as m...

C

Chris Jayden•1/14/25, 9:34 AM

That's interesting. What are the differences?

C

Chris Jayden•1/14/25, 9:35 AM

I see the sqlite is quite a recent thing in DO? https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-storage/#sqlite-storage-backend

C

Chris Jayden•1/14/25, 9:36 AM

This is a helpful blog I just found that helps to explain that use case a bit more; https://www.lambrospetrou.com/articles/durable-objects-cloudflare/

Lambros Petrou personal website

Durable Objects (DO) — Unlimited single-threaded servers spread acr...

Understand Cloudflare Durable Objects and use them to simplify your architecture.

CChris Jayden I see the sqlite is quite a recent thing in DO? https://developers.cloudflare.co...

L

lambrospetrou•1/14/25, 9:40 AM

Yeah, SQLite in DO launched a few months ago, but it has been used by D1 for more than a year now.

If you also want to see how to think about using many DOs check out https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/

I have a real example implementing that pattern explained in https://www.lambrospetrou.com/articles/tiddlyflare/

Also https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-storage/#sql-in-durable-objects-vs-d1 has some light comparison between the two.

C

Chris Jayden•1/14/25, 9:40 AM

Haha i just noticed

C

Chris Jayden•1/14/25, 9:40 AM

that's your blog

C

Chris Jayden•1/14/25, 9:40 AM

what a weird coincidence

C

Chris Jayden•1/14/25, 9:41 AM

these articles are terrific @lambrospetrou

CChris Jayden I just found out about the 10 GB limit on D1. And I think I'll need at least 30-...

D

DevilsOP•1/14/25, 11:20 AM

Personally my stack right now is 3 VPS’s all with around 80gb of storage running YugabyteDB’s..

D

DevilsOP•1/14/25, 11:20 AM

But like lambrospetrou said, it would be much harder to switch or migrate to another database.

D

DevilsOP•1/14/25, 11:21 AM

I’m having to build a loadbalancer and pooler.

D

DevilsOP•1/14/25, 11:21 AM

Even managed databases are eh.

DDevils Personally my stack right now is 3 VPS’s all with around 80gb of storage running...

C

Chris Jayden•1/14/25, 1:21 PM

Appreciate you chiming in. Noted ! i'm really intrigued by Durable Objects. They seem underrated/underappreciated. I thought they were mostly for real-time stuff, but it seems like a great way to scale!

FFriday Candour great to scale? elaborate.

I

Isaac McFadyen•1/14/25, 6:03 PM

You can create unlimited Durable Objects each with their own SQLite database.

I

Isaac McFadyen•1/14/25, 6:08 PM

You're correct, unlimited is never unlimited, but in this case "unlimited" is so high that if you reached that point you'd have an Enterprise agreement with them and could negotiate higher limits anyway.
Using D1 via API is slow because of the architecture, it's not some grand scheme to force you into using a Worker. D1 is located on Cloudflare's edge network but the API routes through the cores (because it was designed well before D1 and that was how it was built).

C

Chaika•1/14/25, 6:11 PM

They run on the same proxy machine that handles your connection

C

Chaika•1/14/25, 6:11 PM

so yes, very edge, about as edge as you can get lol

A

Ashkan•1/14/25, 11:06 PM

Hi Dear CF Team, In the D1 metric tab I think it lacks some informative data, such as what are the top 5 or most read tables (in some situations when an cyber attack happens the most read tables are the clues for security location hole). (feedback)

AAshkan Hi Dear CF Team, In the D1 metric tab I think it lacks some informative data, su...

C

Chaika•1/14/25, 11:09 PM

D1 has the insights command in wrangler: https://developers.cloudflare.com/d1/observability/metrics-analytics/#query-insights, which can show the most executed queries, slowest, reading the most rows/writing the most rows, etc

Cloudflare Docs

Metrics and analytics · Cloudflare D1 docs

D1 exposes database analytics that allow you to inspect query volume, query latency, and storage size across all and/or each database in your account.

CChaika D1 has the insights command in wrangler: https://developers.cloudflare.com/d1/ob...

A

Ashkan•1/14/25, 11:16 PM

Thank you for your reply. it works for developers who are familiar with wrangler and so on.

A

Alex Patterson•1/14/25, 11:18 PM

I have setup a project using

pnpx wrangler d1 create cove

pnpx wrangler d1 create cove

pnpx wrangler d1 create cove

pnpx wrangler d1 create cove It doesn't seem like I can use this with a

--local

--local

flag, so it doesn't seem like we can truly use this as local development only. As in NOTHING created remote.

A

Ashkan•1/14/25, 11:46 PM

@Alex Patterson , If I understood your problem well, in order to create or signal cloudflare to create your database remotely you must use

--remote

--remote

flag when executing your command for example

wrangler d1 execute  YOUR_DB_NAME --file ./database/script.sql --remote

wrangler d1 execute YOUR_DB_NAME --file ./database/script.sql --remote

wrangler d1 execute  YOUR_DB_NAME --file ./database/script.sql --remote

wrangler d1 execute YOUR_DB_NAME --file ./database/script.sql --remote

AAlex Patterson I have setup a project using `pnpx wrangler d1 create cove` It doesn't seem lik...

M

Max (@rozenmd)•1/15/25, 9:26 AM

there's no create step for local dbs, you should be able to just start querying

C

cosbgn•1/15/25, 10:06 AM

I cache my D1 connection like this (on workers):

let cached_db = null;
export const useDb = () => {
  if (!cached_db) {
    const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB
    cached_db = drizzle(binding, { schema })
  }
  return cached_db
}

let cached_db = null;
export const useDb = () => {
  if (!cached_db) {
    const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB
    cached_db = drizzle(binding, { schema })
  }
  return cached_db
}

let cached_db = null;
export const useDb = () => {
  if (!cached_db) {
    const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB
    cached_db = drizzle(binding, { schema })
  }
  return cached_db
}

let cached_db = null;
export const useDb = () => {
  if (!cached_db) {
    const binding = process?.env?.DB || globalThis?.__env__?.DB || globalThis?.DB
    cached_db = drizzle(binding, { schema })
  }
  return cached_db
}

Can this be the cause of:

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

C

cosbgn•1/15/25, 10:06 AM

Should I avoid caching the connection?

Ccosbgn I cache my D1 connection like this (on workers): ```js let cached_db = null; ex...

L

lambrospetrou•1/15/25, 10:08 AM

I don't know how Drizzle works internally, but D1 doesn't have the concept of a long-lived connection like a traditional database. Every query you do sends a request to the database using connections managed by the runtime.
Our recommendation is to always use the fresh

env.{DATABASE_BINDING}

env.{DATABASE_BINDING}

env.{DATABASE_BINDING}

env.{DATABASE_BINDING} you get in each request, otherwise you will definitely hit issues at some point.

This is what their example is doing as well: https://orm.drizzle.team/docs/connect-cloudflare-d1#step-2---initialize-the-driver-and-make-a-query
Looking at https://github.com/drizzle-team/drizzle-orm/blob/main/drizzle-orm/src/d1/driver.ts#L39 it's not doing expensive stuff, so just create it per request.

Llambrospetrou I don't know how Drizzle works internally, but D1 doesn't have the concept of a ...

L

lambrospetrou•1/15/25, 10:10 AM

The specific error you got above is not relevant to that anyway, that error says that your query took too long and was killed.
Check if you are doing any kind of scans over thousands/millions of rows each time.

C

cosbgn•1/15/25, 10:14 AM

Does efficiency 0 means it's really bad?

 "avgRowsRead": 0,
    "totalRowsRead": 13401,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 0.33640454375886003,
    "totalDurationMs": 4508.830100000001,
    "numberOfTimesRun": 13403,
    "queryEfficiency": 0

 "avgRowsRead": 0,
    "totalRowsRead": 13401,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 0.33640454375886003,
    "totalDurationMs": 4508.830100000001,
    "numberOfTimesRun": 13403,
    "queryEfficiency": 0

 "avgRowsRead": 0,
    "totalRowsRead": 13401,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 0.33640454375886003,
    "totalDurationMs": 4508.830100000001,
    "numberOfTimesRun": 13403,
    "queryEfficiency": 0

 "avgRowsRead": 0,
    "totalRowsRead": 13401,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 0.33640454375886003,
    "totalDurationMs": 4508.830100000001,
    "numberOfTimesRun": 13403,
    "queryEfficiency": 0

Ccosbgn Does efficiency 0 means it's really bad? ``` "avgRowsRead": 0, "totalRowsR...

L

lambrospetrou•1/15/25, 10:16 AM

Your query took

0.3ms

0.3ms

0.3ms

0.3ms on average so why are you worried about it?

C

cosbgn•1/15/25, 10:16 AM

mainly because I keep getting this:

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

[request error] [unhandled] [500] D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

C

cosbgn•1/15/25, 10:16 AM

like every day

Ccosbgn mainly because I keep getting this: `[request error] [unhandled] [500] D1_ERROR:...

L

lambrospetrou•1/15/25, 10:17 AM

Then check the query that's failing, what it's doing. Get insights to see your slowest queries: https://developers.cloudflare.com/d1/observability/metrics-analytics/#query-insights

Anyway to cluster multiple D1 Databases together, or would I essentially have to create custom middl

Similar Threads

Similar Threads

Similar Threads