Hey, I recently moved some of our infra from KV to D1, with a database situated in every location to

Hey, I recently moved some of our infra from KV to D1, with a database situated in every location to be nearer to the user while awaiting global replication. This was working great, however after updating that data through our warming process (import all new data into a tmp database, rename old database, rename new database to old database name, drop old database), the databases all appear to be experiencing significant random latency issues. In our monitoring I can see average duration to call D1 jumping from ~20-50ms depending on worker location to some queries hanging for multiple seconds up to a minute. I'm wondering if someone can help me investigate what is going on here. Beyond completely recreating all these databases as part of our warming process I am not sure what the option is here. Our database is ~2GB in size and has 3.24 million rows. I am wondering if there's something going on internally like DO resets that is causing this behaviour. I've attached an example graph where you can see the moment the warming happened and the duration metrics afterwards. This is from the HKG PoP and with a D1 instance in APAC

ZackOP•3/17/25, 4:55 PM

BBoetro Hi! I'm looking for some docs on how d1 handles a colo going doing? If the locat...

lambrospetrou•3/17/25, 6:59 PM

D1 is built ontop of Durable Objects (DO). It's true that once the colo hosting the Durable Object for your database there will be some downtime. But, the Durable Objects platform will move your database DO on a healthy colo/server as soon as the faulty colo is detected by the automation and taken out of the network. Depending on how severe the issue is this move could take seconds or a few minutes. Usually there shouldn't be much of a downtime for this reason.

Unfortunately, most of the errors you will encounter are transient and sporadic, not a whole colo going down. In those cases, making your queries idempotent and having some retries will greatly help.

ZZack Hey, I recently moved some of our infra from KV to D1, with a database situated ...

lambrospetrou•3/17/25, 7:02 PM

You need to provide us database IDs to be able to look anything for you.
Also, some details on volume, queries, how much data are you returning each time, etc are helpful.

This is from the HKG PoP and with a D1 instance in APAC

APAC is a huge region and you could be 100-200ms away from your database unfortunately. Having said that anything more than a few hundreds of ms even in those cases is a bigger unexpected issue, either with the responses getting back to your worker or something else.

Also, I assume you are using the Workers binding right? Not the REST API.

Llambrospetrou D1 is built ontop of Durable Objects (DO). It's true that once the colo hosting ...

Boetro•3/17/25, 7:03 PM

Thanks for the response exactly what I was looking for!

Llambrospetrou You need to provide us database IDs to be able to look anything for you. Also, s...

ZackOP•3/17/25, 7:17 PM

Just heading back to my office, will get those to you shortly. Will throw them in a thread

SSaku75 Hey everyone, just a curious question, that i have been searching for an answer ...

lambrospetrou•3/17/25, 8:44 PM

If you need more than that, it's better to use Durable Objects directly, really. At least, at the current state of D1.

Ludo | Txtx•3/17/25, 10:05 PM

howdy! is it possible to have a worker W1 passing its binding to a Service worker W2?
I'd like to have to have a pool of workers, all bound to different D1 databases, being able to delegate the execution of the same logic to one worker (ideally a durable object) that I can upgrade easily

LLudo | Txtx howdy! is it possible to have a worker W1 passing its binding to a Service worke...

Isaac McFadyen•3/17/25, 10:06 PM

?crossposting

SuperHelpflare•3/17/25, 10:06 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

IIsaac McFadyen ?crossposting

Ludo | Txtx•3/17/25, 10:09 PM

I initially posted in workers-for-platform, which I don't think was the right one\

will smith•3/18/25, 1:43 AM

Any update on this? the db id should be in the metrics link
d1-database

Panoptikym•3/18/25, 2:42 AM

#d1-database

FFjordWarden How do I deal with OLTP workloads when using D1, I know that D1 does not suppor...

Burrito•3/18/25, 11:16 PM

D1 does support transactions: https://developers.cloudflare.com/d1/worker-api/d1-database/#batch

akazwz•3/19/25, 6:33 AM

Why is D1 overloaded? Is this product truly production-ready?

Aakazwz Why is D1 overloaded? Is this product truly production-ready?

lambrospetrou•3/19/25, 8:33 AM

D1 as a whole is not overloaded. Each individual database can be overloaded if you send time-consuming queries or thousands of queries per second.

Kite•3/19/25, 12:17 PM

Hey guys, does full text search or just search in general work for multi-tenant D1?

Kite•3/19/25, 12:17 PM

I'm wondering if it would make sense for applications like multi-vendor ecommerce

KKite Hey guys, does full text search or just search in general work for multi-tenant ...

lambrospetrou•3/19/25, 5:52 PM

D1 (and SQLite Durable Objects) support the FTS5 SQLite extension yes. https://developers.cloudflare.com/d1/sql-api/sql-statements/#supported-sqlite-extensions

anchor•3/20/25, 12:05 AM

Hi everyone,

I’m encountering an issue with my local Cloudflare Worker setup using Miniflare and SQLite. Is there a known issue or extra configuration needed for Miniflare to correctly bind and use a local SQLite file for D1?
How can I confirm that Miniflare is loading the correct database file and not creating an in-memory database?
Are there any additional debug flags or logs that might help diagnose this issue?

Jadugar_Jaggu•3/20/25, 6:08 AM

D1 having issues ?

JJadugar_Jaggu D1 having issues ?

Jadugar_Jaggu•3/20/25, 6:12 AM

Having D1 internal reference error @Community Champion
account id 0292f26a5978816ec7dd8d2838ddda52

Is it global issue?

Unsmart•3/20/25, 6:19 AM

?pings

SuperHelpflare•3/20/25, 6:19 AM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

JJadugar_Jaggu D1 having issues ?

itsmatteomanf•3/20/25, 8:44 AM

that is a relatively common error, it happens sometimes, you need to be able to catch it and retry...

will smith•3/21/25, 3:31 AM

Getting the same huge latency again, any updates? Workers id a3f6e70849e6eb5ca1ebaf6083bc84ba. d1-database

olafg•3/21/25, 1:55 PM

How are you supposed to setup wrangler (jsonc) to have a local development database and a production instance? Wrangler only supports production and preview as environments, but if I define a d1 binding on the top level it takes precedence over the one defined in the environment

Lucas•3/21/25, 4:15 PM

Does Cloudflare D1 support database connection pooling like Supabase or Neon?
If not, does Cloudflare D1 support something similar?
If not, it's not a good option to choose D1 instead Supabase or Neon, right? (also considering other issues like lack of transactions support)
If right, what could be a good purpose to use D1 instead Postgres?

LLucas Does Cloudflare D1 support database connection pooling like Supabase or Neon? If...

Chaika•3/21/25, 4:27 PM

Does Cloudflare D1 support database connection pooling like Supabase or Neon?

D1 is not postgres. It's sqlite, which doesn't have a remote connection protocol. You can connect via the Rest API( not recommended, goes through core/higher latency and api rate limits) or via Workers, which just make requests to your d1 instance.

If not, it's not a good option to choose D1 instead Supabase or Neon, right? (also considering other issues like lack of transactions support)

Supabase and Neon are both Postgres DBs. Compared to D1, they have more scaling options (D1 is locked to a single instance in a single region right now), higher max storage (D1 has a hard limit of 10 gb), and is postgres/instead of sqlite, less vendor lockin and such. So it's pretty different in general. D1 aims to have you scale with many DBs rather then one big one, but lacks the dynamic bindings to make that easy right now

Lucas•3/21/25, 4:28 PM

I know D1 is SQLite.

Chaika•3/21/25, 4:29 PM

You can use neon w/ Workers using Cloudflare's Hyperdrive connecting pooling/caching service: https://neon.tech/docs/guides/cloudflare-hyperdrive, or their connecting pooling

LLucas I know D1 is SQLite.

Chaika•3/21/25, 4:31 PM

Then maybe it helps to extend my answer: Since D1 is SQlite, Sqlite doesn't have external database connections/any connection protocol, there's no expensive db connections to be had. When your query your db from your worker, you're getting the best performance to d1 without having to do anything, under the hood it uses rpc/http, much less expensive connections then db connections, to connect and talk to your database

Lucas•3/21/25, 4:50 PM

@Chaika Ok, thank you.

masimo•3/21/25, 9:20 PM

According to docs:

D1 is designed for horizontal scale out across multiple, smaller (10 GB) databases, such as per-user, per-tenant or per-entity databases.

But I don't really see a way to dynamically bind to a D1 from a worker? How are per-user, per-tenant, or per-entity use cases actually covered? Any examples?

Mmasimo According to docs: > D1 is designed for horizontal scale out across multiple, sm...

Isaac McFadyen•3/21/25, 9:26 PM

There is no way to dynamically bind D1 databases yet, but it's coming in the future at some point.

masimo•3/21/25, 9:35 PM

I see, thank you. So if I understand, the per-user, per-tenant, or per-entity database use cases are covered by manually updating bindings to a worker and redeploying the worker?

masimo•3/21/25, 9:40 PM

Looks like https://developers.cloudflare.com/d1/platform/limits/ says max bindings is approximately 5,000. So there is that constraint as well on the "per-user, per-tenant or per-entity database" use case..

Mmasimo Looks like https://developers.cloudflare.com/d1/platform/limits/ says max bindin...

Isaac McFadyen•3/21/25, 9:41 PM

Correct on both accounts, yeah. Hopefully in the future there will be a workaround/solution for these limitations but none at the moment.

masimo•3/21/25, 9:41 PM

Roger roger, thanks - will keep an eye on how that develops

laury•3/21/25, 9:56 PM

wanting to modify a table of a d1 database using migrations, i added a second SQL file with the alter script. when I apply the migrations by 'wrangler migration apply' It re-apply the first SQL file as well (with the creation script) and obviously i get an errore. how to manage the db changes in a viable and repeatible way

Mmasimo Roger roger, thanks - will keep an eye on how that develops

ajgeiss0702•3/21/25, 9:57 PM

cloudflare has said in the past that dynamic bindings for things like D1 are a goal, but theyve been saying that for a while.

A current solution could be to use Durable Objects' SQL storage API, but note that it has different limits and pricing than D1, and might have different features.
DO's SQL Storage could let you have a seperate object (and hence database) for each user. As a bonus, the DO would be created close to the first request, so each users' database could be located physically closer to the user.

Also note that Cloudflare Developer Week is coming up (first week of april) and they might announce relevant things during that. Don't count on it though.

Llaury wanting to modify a table of a d1 database using migrations, i added a second SQ...

ajgeiss0702•3/21/25, 10:06 PM

I personally don't have any experience with D1's migrations system, but make sure that the migration version numbers are sequential

Aajgeiss0702 I personally don't have any experience with D1's migrations system, but make sur...

laury•3/22/25, 9:55 AM

Yes are sequential but Wrangler seems apply all scripts everytime. It Seems doesn't keep any state about migration

Kalu•3/24/25, 5:01 PM

I have an sqlite database of statistics that I update locally. I'd like to create a site to display certain statistics. Do you think it's usable? D1, is there a way to replace the sqlite file a la voler or do I have to code a warper and run it in a DO? (my data base is around 300MB)

KKalu I have an sqlite database of statistics that I update locally. I'd like to creat...

lambrospetrou•3/24/25, 6:39 PM

I don't know what "a la voler" means, but you can export your local database and import it into a D1 database. Depending on the length of each row statement and the overall query load you might need to split the import into smaller chunks, but doable.

laury•3/24/25, 10:17 PM

Anyone can use D1-STUDIO on windows sucessfully? (https://d1-studio.com)

npm install -g d1-studio
npx d1s

I got this error:

Error in tables-route: Command failed: wrangler d1 execute authdb --local --command="
SELECT name FROM sqlitemaster
WHERE type='table'
AND name NOT LIKE 'sqlite%'
AND name NOT LIKE 'cf%'
ORDER BY name;
"
X [ERROR] Error: must provide --command or --file.

Node 22.14.00

d1-studio

The first native studio for D1 databases

Visal .In•3/25/25, 4:23 AM

Alternatively, you can use

npx @outerbase/studio

npx @outerbase/studio

inside your cloudflare project to view D1 database

Hey, I recently moved some of our infra from KV to D1, with a database situated in every location to

Similar Threads

Similar Threads

Similar Threads