Another question regarding read replicas, mainly because I don't quite understand CF regions - is th

Another question regarding read replicas, mainly because I don't quite understand CF regions - is there only ever one replica per region, or is it more local than that?
Eg: APAC is a pretty large region, if someone from Perth, Australia makes a request would the replica be made there or is it a more central location like Sydney? If a second request comes from Tokyo, would it use the first replica or make a new one nearby?

Mmon Another question regarding read replicas, mainly because I don't quite understan...

Isaac McFadyen•5/8/25, 1:52 AM

Per region. And replicas are created once when you enable them, they won't be created on-demand like you describe in your second scenario.

Isaac McFadyen•5/8/25, 1:52 AM

I.e requests from APAC will go to the single replica in APAC (or whatever the replication regioning system is, I'm not sure it's as clearly defined as that), which is created when you enable replicas initially.

monOP•5/8/25, 2:08 AM

Gotcha, thanks

Mmon Another question regarding read replicas, mainly because I don't quite understan...

Chaika•5/8/25, 2:31 AM

Apac is large but Cloudflare subdivides it. Australia is part of their Oceanic region https://where.durableobjects.live/region/oc and Tokyo would be part of Asia Pacific https://where.durableobjects.live/region/apac
two different read replica locations

Chaika•5/8/25, 2:33 AM

Still what you described can essentially happen with Singapore vs Tokyo for example, with same region. Could have your read replica placed in Singapore even if you only get traffic from Japan. Asia Pacific is def the messiest region

monOP•5/8/25, 2:35 AM

Hmm, when using the "served from" metadata in a response, will it be precise enough to know if it was Sydney vs Tokyo, or will it just return APAC?

Mmon Hmm, when using the "served from" metadata in a response, will it be precise eno...

Chaika•5/8/25, 2:35 AM

just apac

Chaika•5/8/25, 2:36 AM

(Sydney would return OC not APAC)

Chaika•5/8/25, 2:36 AM

Nothing you can do if it's at a suboptimal location within a region eitherway tho

monOP•5/8/25, 2:37 AM

Right right, shame there's no visibility there, though I imagine you do that and people will start asking for precise location hints

Chaika•5/8/25, 2:38 AM

yea, something I don't think CF would ever do as it'd limit their ability to remove locations within a region/more dependent on specific locations
think the more likely solution is just more smart placement within the region

monOP•5/8/25, 3:11 AM

Do you know if the REST api can "leak" the location just by the IP it gets routed to? I can probably just test that myself now...
I'm thinking that smart placement + read replicas are going to be (mostly) a panacea to my performance problems

monOP•5/8/25, 3:12 AM

I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs) is about 2.5 seconds worse, and I'm curious if you're seralising to JSON using a JS thing behind the scenes, which would explain that slowness.

Mmon Do you know if the REST api can "leak" the location just by the IP it gets route...

Chaika•5/8/25, 3:14 AM

hmm by the ip it gets routed to? The Cloudflare Rest API is just behind the normal proxy, and I believe it still sends all queries to one of the cores first (eu, us) making apac queries via REST rather slow

monOP•5/8/25, 3:14 AM

My actual use case is via workers, I'm just making sure it's not my worker code by testing with REST, and seeing very similar timings

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

Chaika•5/8/25, 3:22 AM

Some of its part of the open source runtime: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L159
I believe it's something like d1 worker -> d1 durable object, and Workers RPC calls (which uses V8 Serialization for values and Cap'n proto for the protocol itself)

Chaika•5/8/25, 3:23 AM

I have noticed the same slowness myself with returning lots of data though

Chaika•5/8/25, 3:26 AM

hmm perhaps there's more json then I thought: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L239C1-L239C33
interesting. They mentioned publicly about switching to RPC before, but maybe just the outer layer

monOP•5/8/25, 3:27 AM

That would certainly explain it, it seems to match with the JSON serialisation benchmarks I've seen for v8

Mmon Another question regarding read replicas, mainly because I don't quite understan...

lambrospetrou•5/8/25, 9:18 AM

For now yes, that's correct, one static replica per region. There is certainly a roadmap to extend that and improve it over the next several months to be more dynamic and follow your traffic patterns, but it's still some time away.

Vlad Lykhonis•5/8/25, 8:53 PM

Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size 10GB what are strategies to mitigate and continue to scale large databases?

I’m leaning towards some sharding technique based on ids ranges of some sort. I have not looked into this deeply yet, why I’m curious to hear if anyone already see some more general reliable way to scale above 10Gb.

VVlad Lykhonis Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size ...

1984 Ford Laser•5/8/25, 10:05 PM

Durable Objects

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

monOP•5/9/25, 1:29 AM

So I did some actual benchmarks and this doesn't make sense really (serialisation should take tens of ms). I went back in the discord history, and every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.
Is there any knowledge/work behind the scenes on this? I'd be perfectly happy with "it's [a network layer | unoptimised code | something else] and we know about it" just to get some confidence. This is APAC, either REST or using the Workers SDK, warm DB (seconds between requests) with no concurrent writes during the read, same performance characteristics through both services.

11984 Ford Laser Durable Objects

Vlad Lykhonis•5/9/25, 1:40 AM

What durable objects? How is it helpful.

VVlad Lykhonis What durable objects? How is it helpful.

Isaac McFadyen•5/9/25, 1:45 AM

Each DO can have it's own SQLite database, and you can spawn unlimited number of DOs (within your account-wide limit): https://blog.cloudflare.com/sqlite-in-durable-objects/

Isaac McFadyen•5/9/25, 1:45 AM

https://developers.cloudflare.com/durable-objects/#what-are-durable-objects

IIsaac McFadyen Each DO can have it's own SQLite database, and you can spawn unlimited number of...

Vlad Lykhonis•5/9/25, 5:05 AM

That wasn’t a question. I can create 50k D1s too and extend limit. Question was about what to do once D1 reaches 10GB storage.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

Isaac McFadyen•5/9/25, 5:06 AM

If you create multiple D1 DBs you have to bind them manually to a Worker. A DO can be created and invoked without manually redeploying the Worker.

Isaac McFadyen•5/9/25, 5:07 AM

As for sharding or similar strategies, that's entirely application-specific.

IIsaac McFadyen As for sharding or similar strategies, that's entirely application-specific.

Vlad Lykhonis•5/9/25, 5:17 AM

This was the question.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

lambrospetrou•5/9/25, 7:43 AM

The 10GB is a hard limit so you need to either migrate your data out of the single database if you get there, or plan ahead by using multiple databases or something else like sharding across Durable Objects or Hyperdrive+traditional database.

Mmon So I did some _actual_ benchmarks and this doesn't make sense really (serialisat...

lambrospetrou•5/9/25, 7:46 AM

If you make a clear question I can answer it for you. Your previous question about JSON serialization was answered already linking to the code of the D1 binding which yes, it does JSON parsing for the response from the D1 service.

Our D1 worker doesn't do any kind of intermediate serialization/parsing at all, but the result is JSON serialized from the underlying SQLite before being send out. So, there is a JSON serialization once, and a JSON deserialization inside your worker for the result. Nothing special in that serialization, you will get the same perf as JSON.parse/JSON.stringifyJSON.parse/JSON.stringify.

every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.

Be concrete please. "Latency" is not a single thing. There are millions of reasons why something can be slow. User error, network errors, Cloudflare bugs, and lots of other things.

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

lambrospetrou•5/9/25, 7:50 AM

There is a certain bug we know internally (not specific to D1 but to JSRPC) that could lead to slow network transfers if it's several MBs, but at least from your previous comment it seems that you only respond 1-2MBs?

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

monOP•5/9/25, 7:59 AM

Sorry for muddying the waters with the JSON serialisation, I was stupid and took benchmark data from (what turned out to be) quite an old version of V8, instead of just testing it myself to begin with.

In my case, when I say latency I really mean TTFB of the response. Based on the posts I've seen and my own tests, it's a reasonably common note that larger (where I've seen that described as "10,000 rows" and "some megabytes") queries, while executing within milliseconds, will take multiple seconds extra time to start being delivered to the recipient.
Concretely, when I fetch 100 rows of 16KiB BLOBs, if I do SELECT data ...SELECT data ... the response takes ~3500ms to start arriving. RunningSELECT COUNT(data)SELECT COUNT(data) as a control takes about 700ms.
This is when using the REST API via the Cloudflare dashboard, and looking at Firefox's network timings.
I see similar times in a Worker, which should have clued me in that it wasn't the JSON to begin with. The worker and I are both in OC region, the database in APAC.

Llambrospetrou The 10GB is a hard limit so you need to either migrate your data out of the sing...

Vlad Lykhonis•5/9/25, 7:59 AM

Hyperdrive is interesting. Thanks for the ideas

Mmon Sorry for muddying the waters with the JSON serialisation, I was stupid and took...

lambrospetrou•5/9/25, 8:01 AM

Firstly, using the REST API vs querying through a Worker are completely different flows in reality and the REST API is VERY slow at the moment compared to the worker binding which is a direct call to the underlying database.

Do not use the REST API for any benchmarking, your results are going to be super noisy and unrealistic.

We are in the process of migrating the API to not have those limitations, but it needs a few more weeks before being released publicly.

Llambrospetrou Firstly, using the REST API vs querying through a Worker are completely differen...

lambrospetrou•5/9/25, 8:03 AM

Second, TTFB doesn't really matter in your D1 queries since the response needs to arrive in full inside your worker so that it can be JSON parsed before it's delivered to you. So, I am not sure how you are measuring TTFB. If you give some details on that I might be able to explain it.

monOP•5/9/25, 8:04 AM

I'm definitely not measuring TTFB inside the worker, but the numbers I got were so similar I (incorrectly) assumed that the delay there was a similar cause.

monOP•5/9/25, 8:05 AM

Also, sorry if "never responded to by a cloudflare employee" came across as an attack - was meant to mean that there wasn't an existing explanation I could fall back on.

Mmon I'm definitely not measuring TTFB inside the worker, but the numbers I got were ...

lambrospetrou•5/9/25, 8:13 AM

I can definitely argue that any latency comparison between REST API and Worker Binding is 99% of the time not the same root cause. If you have concrete numbers from your Worker's end to end latency, feel free to post later and I can take a look.
Just mentioning it even if you already know, but each D1 query includes the time it took for the SQL itself to be executed (usually at the single digit millisecond level or less), and then you can measure the end-to-end D1 latency within your worker. Example of measuring that in https://github.com/lambrospetrou/d1-starter-sessions-api/blob/6bbd1e5ee971240bb05b669488818f5976d7f81d/src/index.ts#L87

GitHub

d1-starter-sessions-api/src/index.ts at 6bbd1e5ee971240bb05b6694888...

Starter repository using Cloudflare Workers with D1 database and the new Sessions API for read replication. - lambrospetrou/d1-starter-sessions-api

monOP•5/9/25, 8:15 AM

Sure - I'll come back on Monday with some numbers from that.

Original message was deleted

lambrospetrou•5/9/25, 3:28 PM

That error doesn't refer to your traffic being too much. Usually it's due to either a query generating too many changes and then failing to persist, or the underlying machine has network issues and could not persist your changes (even if small).
Are you seeing this consistently over different times, or did you see it now a few times?
BTW, without a database ID we can't really help much in terms of investigation.

Original message was deleted

lambrospetrou•5/9/25, 3:36 PM

I literally only see a single error for your database, and the location of your database had some sporadic network issue spikes throughout the day, so it seems like you were just unlucky.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

gruntlord6•5/9/25, 4:49 PM

when they eventually making dynamic bindings its a great way to handle this scenario without the complexity of durable objects. Sharding is the way IMO since each db is a single thread anyway

gruntlord6•5/9/25, 4:52 PM

I haven't used this but if it comes up for me and they still don't have the dynamic bindings officially this would be worth investigating: https://discord.com/channels/595317990191398933/992060581832032316/1357236057569099849

Ggruntlord6 I haven't used this but if it comes up for me and they still don't have the dyna...

Vlad Lykhonis•5/9/25, 9:23 PM

I am using workers for platforms, so allocating new D1 and binding it to a my clients scripts should be no problem. I was curious if there is some more general approach for sharding, but indeed seems more situational aka application specifics

gruntlord6•5/9/25, 9:24 PM

yea its always going to come down to what data is suitable to be sharded. You can approach it all kinds of ways. Does each client get a db? each organization? each member of the org? etc

gruntlord6•5/9/25, 9:25 PM

in theory you could have a central db that just maps data from other dbs, that would probably work for most data sets but be more complicated to manage

Ggruntlord6 in theory you could have a central db that just maps data from other dbs, that w...

lambrospetrou•5/10/25, 8:21 AM

Many folks do this, but I recommend against it. If you put a single database (or even Durable Object) in the hot path of every request you lost all the benefits of sharding across multiple of them. If that one DB/DO is down, then you halt.
It's totally OK to have one control plane database for things like managing your resources during creation/deletion/listing, but for the hot path and data plane operations of your application you should not depend on that single DB/DO.
I wrote about this pattern here: https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/

Cloudflare Docs

Control and data plane architectural pattern for Durable Objects

Durable Objects are built on-top of Cloudflare Workers spanning several locations across our global infrastructure network.
Each Durable Object instance has its own durable storage persisted across requests, in-memory state, single-threaded execution, and can be placed in a specific region.

Another question regarding read replicas, mainly because I don't quite understand CF regions - is th

Similar Threads