I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs) is about 2.5 second

I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs) is about 2.5 seconds worse, and I'm curious if you're seralising to JSON using a JS thing behind the scenes, which would explain that slowness.

Mmon Do you know if the REST api can "leak" the location just by the IP it gets route...

Chaika•5/8/25, 3:14 AM

hmm by the ip it gets routed to? The Cloudflare Rest API is just behind the normal proxy, and I believe it still sends all queries to one of the cores first (eu, us) making apac queries via REST rather slow

monOP•5/8/25, 3:14 AM

My actual use case is via workers, I'm just making sure it's not my worker code by testing with REST, and seeing very similar timings

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

Chaika•5/8/25, 3:22 AM

Some of its part of the open source runtime: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L159
I believe it's something like d1 worker -> d1 durable object, and Workers RPC calls (which uses V8 Serialization for values and Cap'n proto for the protocol itself)

Chaika•5/8/25, 3:23 AM

I have noticed the same slowness myself with returning lots of data though

Chaika•5/8/25, 3:26 AM

hmm perhaps there's more json then I thought: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L239C1-L239C33
interesting. They mentioned publicly about switching to RPC before, but maybe just the outer layer

monOP•5/8/25, 3:27 AM

That would certainly explain it, it seems to match with the JSON serialisation benchmarks I've seen for v8

Mmon Another question regarding read replicas, mainly because I don't quite understan...

lambrospetrou•5/8/25, 9:18 AM

For now yes, that's correct, one static replica per region. There is certainly a roadmap to extend that and improve it over the next several months to be more dynamic and follow your traffic patterns, but it's still some time away.

Vlad Lykhonis•5/8/25, 8:53 PM

Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size 10GB what are strategies to mitigate and continue to scale large databases?

I’m leaning towards some sharding technique based on ids ranges of some sort. I have not looked into this deeply yet, why I’m curious to hear if anyone already see some more general reliable way to scale above 10Gb.

VVlad Lykhonis Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size ...

1984 Ford Laser•5/8/25, 10:05 PM

Durable Objects

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

monOP•5/9/25, 1:29 AM

So I did some actual benchmarks and this doesn't make sense really (serialisation should take tens of ms). I went back in the discord history, and every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.
Is there any knowledge/work behind the scenes on this? I'd be perfectly happy with "it's [a network layer | unoptimised code | something else] and we know about it" just to get some confidence. This is APAC, either REST or using the Workers SDK, warm DB (seconds between requests) with no concurrent writes during the read, same performance characteristics through both services.

11984 Ford Laser Durable Objects

Vlad Lykhonis•5/9/25, 1:40 AM

What durable objects? How is it helpful.

VVlad Lykhonis What durable objects? How is it helpful.

Isaac McFadyen•5/9/25, 1:45 AM

Each DO can have it's own SQLite database, and you can spawn unlimited number of DOs (within your account-wide limit): https://blog.cloudflare.com/sqlite-in-durable-objects/

Isaac McFadyen•5/9/25, 1:45 AM

https://developers.cloudflare.com/durable-objects/#what-are-durable-objects

IIsaac McFadyen Each DO can have it's own SQLite database, and you can spawn unlimited number of...

Vlad Lykhonis•5/9/25, 5:05 AM

That wasn’t a question. I can create 50k D1s too and extend limit. Question was about what to do once D1 reaches 10GB storage.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

Isaac McFadyen•5/9/25, 5:06 AM

If you create multiple D1 DBs you have to bind them manually to a Worker. A DO can be created and invoked without manually redeploying the Worker.

Isaac McFadyen•5/9/25, 5:07 AM

As for sharding or similar strategies, that's entirely application-specific.

IIsaac McFadyen As for sharding or similar strategies, that's entirely application-specific.

Vlad Lykhonis•5/9/25, 5:17 AM

This was the question.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

lambrospetrou•5/9/25, 7:43 AM

The 10GB is a hard limit so you need to either migrate your data out of the single database if you get there, or plan ahead by using multiple databases or something else like sharding across Durable Objects or Hyperdrive+traditional database.

Mmon So I did some _actual_ benchmarks and this doesn't make sense really (serialisat...

lambrospetrou•5/9/25, 7:46 AM

If you make a clear question I can answer it for you. Your previous question about JSON serialization was answered already linking to the code of the D1 binding which yes, it does JSON parsing for the response from the D1 service.

Our D1 worker doesn't do any kind of intermediate serialization/parsing at all, but the result is JSON serialized from the underlying SQLite before being send out. So, there is a JSON serialization once, and a JSON deserialization inside your worker for the result. Nothing special in that serialization, you will get the same perf as

JSON.parse/JSON.stringify

JSON.parse/JSON.stringify

every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.

Be concrete please. "Latency" is not a single thing. There are millions of reasons why something can be slow. User error, network errors, Cloudflare bugs, and lots of other things.

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

lambrospetrou•5/9/25, 7:50 AM

There is a certain bug we know internally (not specific to D1 but to JSRPC) that could lead to slow network transfers if it's several MBs, but at least from your previous comment it seems that you only respond 1-2MBs?

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

monOP•5/9/25, 7:59 AM

Sorry for muddying the waters with the JSON serialisation, I was stupid and took benchmark data from (what turned out to be) quite an old version of V8, instead of just testing it myself to begin with.

In my case, when I say latency I really mean TTFB of the response. Based on the posts I've seen and my own tests, it's a reasonably common note that larger (where I've seen that described as "10,000 rows" and "some megabytes") queries, while executing within milliseconds, will take multiple seconds extra time to start being delivered to the recipient.
Concretely, when I fetch 100 rows of 16KiB BLOBs, if I do

SELECT data ...

SELECT data ...

the response takes ~3500ms to start arriving. Running

SELECT COUNT(data)

SELECT COUNT(data)

as a control takes about 700ms.
This is when using the REST API via the Cloudflare dashboard, and looking at Firefox's network timings.
I see similar times in a Worker, which should have clued me in that it wasn't the JSON to begin with. The worker and I are both in OC region, the database in APAC.

Llambrospetrou The 10GB is a hard limit so you need to either migrate your data out of the sing...

Vlad Lykhonis•5/9/25, 7:59 AM

Hyperdrive is interesting. Thanks for the ideas

Mmon Sorry for muddying the waters with the JSON serialisation, I was stupid and took...

lambrospetrou•5/9/25, 8:01 AM

Firstly, using the REST API vs querying through a Worker are completely different flows in reality and the REST API is VERY slow at the moment compared to the worker binding which is a direct call to the underlying database.

Do not use the REST API for any benchmarking, your results are going to be super noisy and unrealistic.

We are in the process of migrating the API to not have those limitations, but it needs a few more weeks before being released publicly.

Llambrospetrou Firstly, using the REST API vs querying through a Worker are completely differen...

lambrospetrou•5/9/25, 8:03 AM

Second, TTFB doesn't really matter in your D1 queries since the response needs to arrive in full inside your worker so that it can be JSON parsed before it's delivered to you. So, I am not sure how you are measuring TTFB. If you give some details on that I might be able to explain it.

monOP•5/9/25, 8:04 AM

I'm definitely not measuring TTFB inside the worker, but the numbers I got were so similar I (incorrectly) assumed that the delay there was a similar cause.

monOP•5/9/25, 8:05 AM

Also, sorry if "never responded to by a cloudflare employee" came across as an attack - was meant to mean that there wasn't an existing explanation I could fall back on.

Mmon I'm definitely not measuring TTFB inside the worker, but the numbers I got were ...

lambrospetrou•5/9/25, 8:13 AM

I can definitely argue that any latency comparison between REST API and Worker Binding is 99% of the time not the same root cause. If you have concrete numbers from your Worker's end to end latency, feel free to post later and I can take a look.
Just mentioning it even if you already know, but each D1 query includes the time it took for the SQL itself to be executed (usually at the single digit millisecond level or less), and then you can measure the end-to-end D1 latency within your worker. Example of measuring that in https://github.com/lambrospetrou/d1-starter-sessions-api/blob/6bbd1e5ee971240bb05b669488818f5976d7f81d/src/index.ts#L87

GitHub

d1-starter-sessions-api/src/index.ts at 6bbd1e5ee971240bb05b6694888...

Starter repository using Cloudflare Workers with D1 database and the new Sessions API for read replication. - lambrospetrou/d1-starter-sessions-api

monOP•5/9/25, 8:15 AM

Sure - I'll come back on Monday with some numbers from that.

Original message was deleted

lambrospetrou•5/9/25, 3:28 PM

That error doesn't refer to your traffic being too much. Usually it's due to either a query generating too many changes and then failing to persist, or the underlying machine has network issues and could not persist your changes (even if small).
Are you seeing this consistently over different times, or did you see it now a few times?
BTW, without a database ID we can't really help much in terms of investigation.

Original message was deleted

lambrospetrou•5/9/25, 3:36 PM

I literally only see a single error for your database, and the location of your database had some sporadic network issue spikes throughout the day, so it seems like you were just unlucky.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

gruntlord6•5/9/25, 4:49 PM

when they eventually making dynamic bindings its a great way to handle this scenario without the complexity of durable objects. Sharding is the way IMO since each db is a single thread anyway

gruntlord6•5/9/25, 4:52 PM

I haven't used this but if it comes up for me and they still don't have the dynamic bindings officially this would be worth investigating: https://discord.com/channels/595317990191398933/992060581832032316/1357236057569099849

Ggruntlord6 I haven't used this but if it comes up for me and they still don't have the dyna...

Vlad Lykhonis•5/9/25, 9:23 PM

I am using workers for platforms, so allocating new D1 and binding it to a my clients scripts should be no problem. I was curious if there is some more general approach for sharding, but indeed seems more situational aka application specifics

gruntlord6•5/9/25, 9:24 PM

yea its always going to come down to what data is suitable to be sharded. You can approach it all kinds of ways. Does each client get a db? each organization? each member of the org? etc

gruntlord6•5/9/25, 9:25 PM

in theory you could have a central db that just maps data from other dbs, that would probably work for most data sets but be more complicated to manage

Ggruntlord6 in theory you could have a central db that just maps data from other dbs, that w...

lambrospetrou•5/10/25, 8:21 AM

Many folks do this, but I recommend against it. If you put a single database (or even Durable Object) in the hot path of every request you lost all the benefits of sharding across multiple of them. If that one DB/DO is down, then you halt.
It's totally OK to have one control plane database for things like managing your resources during creation/deletion/listing, but for the hot path and data plane operations of your application you should not depend on that single DB/DO.
I wrote about this pattern here: https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/

Cloudflare Docs

Control and data plane architectural pattern for Durable Objects

Durable Objects are built on-top of Cloudflare Workers spanning several locations across our global infrastructure network.
Each Durable Object instance has its own durable storage persisted across requests, in-memory state, single-threaded execution, and can be placed in a specific region.

will smith•5/11/25, 2:42 PM

is it me or is database editing in the UI broken?
It's straight up editing the wrong row. Video attached
https://dash.cloudflare.com/a3f6e70849e6eb5ca1ebaf6083bc84ba/workers/d1/databases/79a7d7de-4213-4a3a-b53c-e9ac3c1eda91/tables/Page?page=1&perPage=10

Wwill smith is it me or is database editing in the UI broken? It's straight up editing the w...

Brayden•5/11/25, 3:43 PM

If you go back to the D1 resource list page and click the “…” menu, then “Explore data”, try using this new experience & let me know if it works.

BBrayden If you go back to the D1 resource list page and click the “…” menu, then “Explor...

will smith•5/11/25, 4:05 PM

yep this one's working thanks

BBrayden If you go back to the D1 resource list page and click the “…” menu, then “Explor...

gruntlord6•5/11/25, 10:46 PM

that looks a lot nicer, any chance it will have sortable columns like the other version does?

Ggruntlord6 that looks a lot nicer, any chance it will have sortable columns like the other ...

Brayden•5/11/25, 11:54 PM

This week, stay tuned

Llambrospetrou I can definitely argue that any latency comparison between REST API and Worker B...

monOP•5/12/25, 1:43 AM

Here's some typical numbers for small and large returns, where I take a

performance.now()

performance.now()

before and after the call to

bind(...).all()

bind(...).all()

to identify the total time.

row fetch took 206 ms
result serialises to 19906 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 0.4254
  },
  "duration": 0.4254,
  "changes": 0,
  "last_row_id": 705,
  "changed_db": false,
  "size_after": 10797056,
  "rows_read": 2,
  "rows_written": 0
}

row fetch took 3748 ms
result serialises to 5100116 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 2.4634
  },
  "duration": 2.4634,
  "changes": 0,
  "last_row_id": 840,
  "changed_db": false,
  "size_after": 13021184,
  "rows_read": 100,
  "rows_written": 0
}

row fetch took 206 ms
result serialises to 19906 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 0.4254
  },
  "duration": 0.4254,
  "changes": 0,
  "last_row_id": 705,
  "changed_db": false,
  "size_after": 10797056,
  "rows_read": 2,
  "rows_written": 0
}

row fetch took 3748 ms
result serialises to 5100116 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 2.4634
  },
  "duration": 2.4634,
  "changes": 0,
  "last_row_id": 840,
  "changed_db": false,
  "size_after": 13021184,
  "rows_read": 100,
  "rows_written": 0
}

Mmon Here's some typical numbers for small and large returns, where I take a `perform...

lambrospetrou•5/12/25, 6:33 AM

Your worker doing this is in APAC too? Judging from the fast response my guess would be that even if it is, probably far away from the DB still.

Llambrospetrou Your worker doing this is in APAC too? Judging from the fast response my guess w...

monOP•5/12/25, 6:36 AM

I'm in OC so I assume the worker will be too - similar timings for deployed worker vs

wrangler dev --remote

wrangler dev --remote

monOP•5/12/25, 6:36 AM

No smart placement

Mmon I'm in OC so I assume the worker will be too - similar timings for deployed work...

lambrospetrou•5/12/25, 6:38 AM

OK OC to APAC is super varied unfortunately. Without digging into it, my guess is that the 5MB response size hits the limitation we know about the JSRPC protocol (not D1 specific), and gets worse when over OC->APAC. I will raise it again with the team since it's been in discussion for some time.

monOP•5/12/25, 6:39 AM

Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably try and migrate this code to Durable Objects to avoid the cross-region shenanigans - I've been told we tried and failed before, but with SQLite in DO now I think it'll be smooth sailing.

monOP•5/12/25, 6:43 AM

Well, not perfectly smooth sailing because

[WARNING] SQLite in Durable Objects is only supported in local mode.

[WARNING] SQLite in Durable Objects is only supported in local mode.

, but I can probably work around that for the time being

Mmon Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably...

lambrospetrou•5/12/25, 6:45 AM

BTW, the limit I mentioned applies to DOs as well, but at least there your code is near the database so you can do your main logic there to reduce the 5MB result further down before returning it out of the DO.

row fetch took 206 ms result serialises to 19906 bytes of JSON result meta { "served_by": "v3-prod", "served_by_region": "APAC", "served_by_primary": true, "timings": { "sql_duration_ms": 0.4254 }, "duration": 0.4254, "changes": 0, "last_row_id": 705, "changed_db": false, "size_after": 10797056, "rows_read": 2, "rows_written": 0 } row fetch took 3748 ms result serialises to 5100116 bytes of JSON result meta { "served_by": "v3-prod", "served_by_region": "APAC", "served_by_primary": true, "timings": { "sql_duration_ms": 2.4634 }, "duration": 2.4634, "changes": 0, "last_row_id": 840, "changed_db": false, "size_after": 13021184, "rows_read": 100, "rows_written": 0 }

I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs) is about 2.5 second

Similar Threads

Similar Threads

Similar Threads