In order to avoid the 1m lag with KV propagating to all regions, I'm thinking to migrate everything

In order to avoid the 1m lag with KV propagating to all regions, I'm thinking to migrate everything into D1. It seems as though D1 is also cheaper for read/writes/storage. Aside from the difference in performance, is there any reason to stick with KV?

Ddaycompacted In order to avoid the 1m lag with KV propagating to all regions, I'm thinking to...

lambrospetrou•5/6/25, 5:29 PM

Scaling and throughput properties of KV and D1 are wildly different. D1 is a single Durable Object in a single location. KV is cached globally after first access.
Very highly dependent on your workload, requests per second, reads vs writes, locations you access it from, cachability, how much data, etc.

mon•5/7/25, 4:40 AM

Sanity check: wrangler no longer has the

--experimental-backend

--experimental-backend

flag, and wrangler d1 infowrangler d1 info no longer returns the version. Have all beta DBs been silently upgraded to the new backend?
Just trying to remove it as an explanation for some poor performance.

nejc•5/7/25, 5:15 AM

hi all, not sure if this is the correct channel, but would it be possible to add filter for max_databaseSizeBytes https://api.cloudflare.com/client/v4/graphql for d1StorageAdaptiveGroups?

Mmon Sanity check: wrangler no longer has the `--experimental-backend` flag, and `wra...

lambrospetrou•5/7/25, 6:02 AM

Do you mean "alpha" databases? We did not upgrade alpha databases to the production version, and no plans to do so. Alpha databases are deprecated and should be rejecting queries.
I will report the missing versionversion info and will fix.

Llambrospetrou Do you mean "alpha" databases? We did not upgrade alpha databases to the product...

mon•5/7/25, 6:10 AM

Excellent, thanks!

Llambrospetrou Do you mean "alpha" databases? We did not upgrade alpha databases to the product...

lambrospetrou•5/7/25, 6:24 AM

I just checked the wrangler code, and the versionversion is shown in wrangler d1 infowrangler d1 info only if it's an alpha database. So, since you are not seeing it your database is not alpha. If you send me the database ID (DM) I can check its version and confirm to you.

Llambrospetrou I just checked the wrangler code, and the `version` is shown in `wrangler d1 inf...

mon•5/7/25, 6:28 AM

I'm happy with that conclusion - I'll try other performance avenues!

mon•5/8/25, 12:59 AM

Another question regarding read replicas, mainly because I don't quite understand CF regions - is there only ever one replica per region, or is it more local than that?
Eg: APAC is a pretty large region, if someone from Perth, Australia makes a request would the replica be made there or is it a more central location like Sydney? If a second request comes from Tokyo, would it use the first replica or make a new one nearby?

Mmon Another question regarding read replicas, mainly because I don't quite understan...

Isaac McFadyen•5/8/25, 1:52 AM

Per region. And replicas are created once when you enable them, they won't be created on-demand like you describe in your second scenario.

Isaac McFadyen•5/8/25, 1:52 AM

I.e requests from APAC will go to the single replica in APAC (or whatever the replication regioning system is, I'm not sure it's as clearly defined as that), which is created when you enable replicas initially.

mon•5/8/25, 2:08 AM

Gotcha, thanks

Mmon Another question regarding read replicas, mainly because I don't quite understan...

Chaika•5/8/25, 2:31 AM

Apac is large but Cloudflare subdivides it. Australia is part of their Oceanic region https://where.durableobjects.live/region/oc and Tokyo would be part of Asia Pacific https://where.durableobjects.live/region/apac
two different read replica locations

Chaika•5/8/25, 2:33 AM

Still what you described can essentially happen with Singapore vs Tokyo for example, with same region. Could have your read replica placed in Singapore even if you only get traffic from Japan. Asia Pacific is def the messiest region

mon•5/8/25, 2:35 AM

Hmm, when using the "served from" metadata in a response, will it be precise enough to know if it was Sydney vs Tokyo, or will it just return APAC?

Mmon Hmm, when using the "served from" metadata in a response, will it be precise eno...

Chaika•5/8/25, 2:35 AM

just apac

Chaika•5/8/25, 2:36 AM

(Sydney would return OC not APAC)

Chaika•5/8/25, 2:36 AM

Nothing you can do if it's at a suboptimal location within a region eitherway tho

mon•5/8/25, 2:37 AM

Right right, shame there's no visibility there, though I imagine you do that and people will start asking for precise location hints

Chaika•5/8/25, 2:38 AM

yea, something I don't think CF would ever do as it'd limit their ability to remove locations within a region/more dependent on specific locations
think the more likely solution is just more smart placement within the region

mon•5/8/25, 3:11 AM

Do you know if the REST api can "leak" the location just by the IP it gets routed to? I can probably just test that myself now...
I'm thinking that smart placement + read replicas are going to be (mostly) a panacea to my performance problems

mon•5/8/25, 3:12 AM

I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs) is about 2.5 seconds worse, and I'm curious if you're seralising to JSON using a JS thing behind the scenes, which would explain that slowness.

Mmon Do you know if the REST api can "leak" the location just by the IP it gets route...

Chaika•5/8/25, 3:14 AM

hmm by the ip it gets routed to? The Cloudflare Rest API is just behind the normal proxy, and I believe it still sends all queries to one of the cores first (eu, us) making apac queries via REST rather slow

mon•5/8/25, 3:14 AM

My actual use case is via workers, I'm just making sure it's not my worker code by testing with REST, and seeing very similar timings

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

Chaika•5/8/25, 3:22 AM

Some of its part of the open source runtime: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L159
I believe it's something like d1 worker -> d1 durable object, and Workers RPC calls (which uses V8 Serialization for values and Cap'n proto for the protocol itself)

Chaika•5/8/25, 3:23 AM

I have noticed the same slowness myself with returning lots of data though

Chaika•5/8/25, 3:26 AM

hmm perhaps there's more json then I thought: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L239C1-L239C33
interesting. They mentioned publicly about switching to RPC before, but maybe just the outer layer

mon•5/8/25, 3:27 AM

That would certainly explain it, it seems to match with the JSON serialisation benchmarks I've seen for v8

Mmon Another question regarding read replicas, mainly because I don't quite understan...

lambrospetrou•5/8/25, 9:18 AM

For now yes, that's correct, one static replica per region. There is certainly a roadmap to extend that and improve it over the next several months to be more dynamic and follow your traffic patterns, but it's still some time away.

Vlad Lykhonis•5/8/25, 8:53 PM

Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size 10GB what are strategies to mitigate and continue to scale large databases?

I’m leaning towards some sharding technique based on ids ranges of some sort. I have not looked into this deeply yet, why I’m curious to hear if anyone already see some more general reliable way to scale above 10Gb.

VVlad Lykhonis Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size ...

1984 Ford Laser•5/8/25, 10:05 PM

Durable Objects

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

mon•5/9/25, 1:29 AM

So I did some actual benchmarks and this doesn't make sense really (serialisation should take tens of ms). I went back in the discord history, and every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.
Is there any knowledge/work behind the scenes on this? I'd be perfectly happy with "it's [a network layer | unoptimised code | something else] and we know about it" just to get some confidence. This is APAC, either REST or using the Workers SDK, warm DB (seconds between requests) with no concurrent writes during the read, same performance characteristics through both services.

11984 Ford Laser Durable Objects

Vlad Lykhonis•5/9/25, 1:40 AM

What durable objects? How is it helpful.

VVlad Lykhonis What durable objects? How is it helpful.

Isaac McFadyen•5/9/25, 1:45 AM

Each DO can have it's own SQLite database, and you can spawn unlimited number of DOs (within your account-wide limit): https://blog.cloudflare.com/sqlite-in-durable-objects/

Isaac McFadyen•5/9/25, 1:45 AM

https://developers.cloudflare.com/durable-objects/#what-are-durable-objects

IIsaac McFadyen Each DO can have it's own SQLite database, and you can spawn unlimited number of...

Vlad Lykhonis•5/9/25, 5:05 AM

That wasn’t a question. I can create 50k D1s too and extend limit. Question was about what to do once D1 reaches 10GB storage.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

Isaac McFadyen•5/9/25, 5:06 AM

If you create multiple D1 DBs you have to bind them manually to a Worker. A DO can be created and invoked without manually redeploying the Worker.

Isaac McFadyen•5/9/25, 5:07 AM

As for sharding or similar strategies, that's entirely application-specific.

IIsaac McFadyen As for sharding or similar strategies, that's entirely application-specific.

Vlad Lykhonis•5/9/25, 5:17 AM

This was the question.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

lambrospetrou•5/9/25, 7:43 AM

The 10GB is a hard limit so you need to either migrate your data out of the single database if you get there, or plan ahead by using multiple databases or something else like sharding across Durable Objects or Hyperdrive+traditional database.

Mmon So I did some _actual_ benchmarks and this doesn't make sense really (serialisat...

lambrospetrou•5/9/25, 7:46 AM

If you make a clear question I can answer it for you. Your previous question about JSON serialization was answered already linking to the code of the D1 binding which yes, it does JSON parsing for the response from the D1 service.

Our D1 worker doesn't do any kind of intermediate serialization/parsing at all, but the result is JSON serialized from the underlying SQLite before being send out. So, there is a JSON serialization once, and a JSON deserialization inside your worker for the result. Nothing special in that serialization, you will get the same perf as JSON.parse/JSON.stringifyJSON.parse/JSON.stringify.

every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.

Be concrete please. "Latency" is not a single thing. There are millions of reasons why something can be slow. User error, network errors, Cloudflare bugs, and lots of other things.

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

lambrospetrou•5/9/25, 7:50 AM

There is a certain bug we know internally (not specific to D1 but to JSRPC) that could lead to slow network transfers if it's several MBs, but at least from your previous comment it seems that you only respond 1-2MBs?

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

mon•5/9/25, 7:59 AM

Sorry for muddying the waters with the JSON serialisation, I was stupid and took benchmark data from (what turned out to be) quite an old version of V8, instead of just testing it myself to begin with.

In my case, when I say latency I really mean TTFB of the response. Based on the posts I've seen and my own tests, it's a reasonably common note that larger (where I've seen that described as "10,000 rows" and "some megabytes") queries, while executing within milliseconds, will take multiple seconds extra time to start being delivered to the recipient.
Concretely, when I fetch 100 rows of 16KiB BLOBs, if I do SELECT data ...SELECT data ... the response takes ~3500ms to start arriving. RunningSELECT COUNT(data)SELECT COUNT(data) as a control takes about 700ms.
This is when using the REST API via the Cloudflare dashboard, and looking at Firefox's network timings.
I see similar times in a Worker, which should have clued me in that it wasn't the JSON to begin with. The worker and I are both in OC region, the database in APAC.

Llambrospetrou The 10GB is a hard limit so you need to either migrate your data out of the sing...

Vlad Lykhonis•5/9/25, 7:59 AM

Hyperdrive is interesting. Thanks for the ideas

Mmon Sorry for muddying the waters with the JSON serialisation, I was stupid and took...

lambrospetrou•5/9/25, 8:01 AM

Firstly, using the REST API vs querying through a Worker are completely different flows in reality and the REST API is VERY slow at the moment compared to the worker binding which is a direct call to the underlying database.

Do not use the REST API for any benchmarking, your results are going to be super noisy and unrealistic.

We are in the process of migrating the API to not have those limitations, but it needs a few more weeks before being released publicly.

Llambrospetrou Firstly, using the REST API vs querying through a Worker are completely differen...

lambrospetrou•5/9/25, 8:03 AM

Second, TTFB doesn't really matter in your D1 queries since the response needs to arrive in full inside your worker so that it can be JSON parsed before it's delivered to you. So, I am not sure how you are measuring TTFB. If you give some details on that I might be able to explain it.

mon•5/9/25, 8:04 AM

I'm definitely not measuring TTFB inside the worker, but the numbers I got were so similar I (incorrectly) assumed that the delay there was a similar cause.

mon•5/9/25, 8:05 AM

Also, sorry if "never responded to by a cloudflare employee" came across as an attack - was meant to mean that there wasn't an existing explanation I could fall back on.

Mmon I'm definitely not measuring TTFB inside the worker, but the numbers I got were ...

lambrospetrou•5/9/25, 8:13 AM

I can definitely argue that any latency comparison between REST API and Worker Binding is 99% of the time not the same root cause. If you have concrete numbers from your Worker's end to end latency, feel free to post later and I can take a look.
Just mentioning it even if you already know, but each D1 query includes the time it took for the SQL itself to be executed (usually at the single digit millisecond level or less), and then you can measure the end-to-end D1 latency within your worker. Example of measuring that in https://github.com/lambrospetrou/d1-starter-sessions-api/blob/6bbd1e5ee971240bb05b669488818f5976d7f81d/src/index.ts#L87

GitHub

d1-starter-sessions-api/src/index.ts at 6bbd1e5ee971240bb05b6694888...

Starter repository using Cloudflare Workers with D1 database and the new Sessions API for read replication. - lambrospetrou/d1-starter-sessions-api

mon•5/9/25, 8:15 AM

Sure - I'll come back on Monday with some numbers from that.

Original message was deleted

lambrospetrou•5/9/25, 3:28 PM

That error doesn't refer to your traffic being too much. Usually it's due to either a query generating too many changes and then failing to persist, or the underlying machine has network issues and could not persist your changes (even if small).
Are you seeing this consistently over different times, or did you see it now a few times?
BTW, without a database ID we can't really help much in terms of investigation.

In order to avoid the 1m lag with KV propagating to all regions, I'm thinking to migrate everything

Similar Threads

Similar Threads

Similar Threads