Durable Objects

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

mon•5/9/25, 1:29 AM

So I did some actual benchmarks and this doesn't make sense really (serialisation should take tens of ms). I went back in the discord history, and every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.
Is there any knowledge/work behind the scenes on this? I'd be perfectly happy with "it's [a network layer | unoptimised code | something else] and we know about it" just to get some confidence. This is APAC, either REST or using the Workers SDK, warm DB (seconds between requests) with no concurrent writes during the read, same performance characteristics through both services.

11984 Ford Laser Durable Objects

Vlad Lykhonis•5/9/25, 1:40 AM

What durable objects? How is it helpful.

VVlad Lykhonis What durable objects? How is it helpful.

Isaac McFadyen•5/9/25, 1:45 AM

Each DO can have it's own SQLite database, and you can spawn unlimited number of DOs (within your account-wide limit): https://blog.cloudflare.com/sqlite-in-durable-objects/

Isaac McFadyen•5/9/25, 1:45 AM

https://developers.cloudflare.com/durable-objects/#what-are-durable-objects

IIsaac McFadyen Each DO can have it's own SQLite database, and you can spawn unlimited number of...

Vlad Lykhonis•5/9/25, 5:05 AM

That wasn’t a question. I can create 50k D1s too and extend limit. Question was about what to do once D1 reaches 10GB storage.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

Isaac McFadyen•5/9/25, 5:06 AM

If you create multiple D1 DBs you have to bind them manually to a Worker. A DO can be created and invoked without manually redeploying the Worker.

Isaac McFadyen•5/9/25, 5:07 AM

As for sharding or similar strategies, that's entirely application-specific.

IIsaac McFadyen As for sharding or similar strategies, that's entirely application-specific.

Vlad Lykhonis•5/9/25, 5:17 AM

This was the question.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

lambrospetrou•5/9/25, 7:43 AM

The 10GB is a hard limit so you need to either migrate your data out of the single database if you get there, or plan ahead by using multiple databases or something else like sharding across Durable Objects or Hyperdrive+traditional database.

Mmon So I did some _actual_ benchmarks and this doesn't make sense really (serialisat...

lambrospetrou•5/9/25, 7:46 AM

If you make a clear question I can answer it for you. Your previous question about JSON serialization was answered already linking to the code of the D1 binding which yes, it does JSON parsing for the response from the D1 service.

Our D1 worker doesn't do any kind of intermediate serialization/parsing at all, but the result is JSON serialized from the underlying SQLite before being send out. So, there is a JSON serialization once, and a JSON deserialization inside your worker for the result. Nothing special in that serialization, you will get the same perf as

JSON.parse/JSON.stringify

JSON.parse/JSON.stringify

every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.

Be concrete please. "Latency" is not a single thing. There are millions of reasons why something can be slow. User error, network errors, Cloudflare bugs, and lots of other things.

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

lambrospetrou•5/9/25, 7:50 AM

There is a certain bug we know internally (not specific to D1 but to JSRPC) that could lead to slow network transfers if it's several MBs, but at least from your previous comment it seems that you only respond 1-2MBs?

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

mon•5/9/25, 7:59 AM

Sorry for muddying the waters with the JSON serialisation, I was stupid and took benchmark data from (what turned out to be) quite an old version of V8, instead of just testing it myself to begin with.

In my case, when I say latency I really mean TTFB of the response. Based on the posts I've seen and my own tests, it's a reasonably common note that larger (where I've seen that described as "10,000 rows" and "some megabytes") queries, while executing within milliseconds, will take multiple seconds extra time to start being delivered to the recipient.
Concretely, when I fetch 100 rows of 16KiB BLOBs, if I do

SELECT data ...

SELECT data ...

the response takes ~3500ms to start arriving. Running

SELECT COUNT(data)

SELECT COUNT(data)

as a control takes about 700ms.
This is when using the REST API via the Cloudflare dashboard, and looking at Firefox's network timings.
I see similar times in a Worker, which should have clued me in that it wasn't the JSON to begin with. The worker and I are both in OC region, the database in APAC.

Llambrospetrou The 10GB is a hard limit so you need to either migrate your data out of the sing...

Vlad Lykhonis•5/9/25, 7:59 AM

Hyperdrive is interesting. Thanks for the ideas

Mmon Sorry for muddying the waters with the JSON serialisation, I was stupid and took...

lambrospetrou•5/9/25, 8:01 AM

Firstly, using the REST API vs querying through a Worker are completely different flows in reality and the REST API is VERY slow at the moment compared to the worker binding which is a direct call to the underlying database.

Do not use the REST API for any benchmarking, your results are going to be super noisy and unrealistic.

We are in the process of migrating the API to not have those limitations, but it needs a few more weeks before being released publicly.

Llambrospetrou Firstly, using the REST API vs querying through a Worker are completely differen...

lambrospetrou•5/9/25, 8:03 AM

Second, TTFB doesn't really matter in your D1 queries since the response needs to arrive in full inside your worker so that it can be JSON parsed before it's delivered to you. So, I am not sure how you are measuring TTFB. If you give some details on that I might be able to explain it.

mon•5/9/25, 8:04 AM

I'm definitely not measuring TTFB inside the worker, but the numbers I got were so similar I (incorrectly) assumed that the delay there was a similar cause.

mon•5/9/25, 8:05 AM

Also, sorry if "never responded to by a cloudflare employee" came across as an attack - was meant to mean that there wasn't an existing explanation I could fall back on.

Mmon I'm definitely not measuring TTFB inside the worker, but the numbers I got were ...

lambrospetrou•5/9/25, 8:13 AM

I can definitely argue that any latency comparison between REST API and Worker Binding is 99% of the time not the same root cause. If you have concrete numbers from your Worker's end to end latency, feel free to post later and I can take a look.
Just mentioning it even if you already know, but each D1 query includes the time it took for the SQL itself to be executed (usually at the single digit millisecond level or less), and then you can measure the end-to-end D1 latency within your worker. Example of measuring that in https://github.com/lambrospetrou/d1-starter-sessions-api/blob/6bbd1e5ee971240bb05b669488818f5976d7f81d/src/index.ts#L87

GitHub

d1-starter-sessions-api/src/index.ts at 6bbd1e5ee971240bb05b6694888...

Starter repository using Cloudflare Workers with D1 database and the new Sessions API for read replication. - lambrospetrou/d1-starter-sessions-api

mon•5/9/25, 8:15 AM

Sure - I'll come back on Monday with some numbers from that.

Original message was deleted

lambrospetrou•5/9/25, 3:28 PM

That error doesn't refer to your traffic being too much. Usually it's due to either a query generating too many changes and then failing to persist, or the underlying machine has network issues and could not persist your changes (even if small).
Are you seeing this consistently over different times, or did you see it now a few times?
BTW, without a database ID we can't really help much in terms of investigation.

Original message was deleted

lambrospetrou•5/9/25, 3:36 PM

I literally only see a single error for your database, and the location of your database had some sporadic network issue spikes throughout the day, so it seems like you were just unlucky.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

gruntlord6•5/9/25, 4:49 PM

when they eventually making dynamic bindings its a great way to handle this scenario without the complexity of durable objects. Sharding is the way IMO since each db is a single thread anyway

gruntlord6•5/9/25, 4:52 PM

I haven't used this but if it comes up for me and they still don't have the dynamic bindings officially this would be worth investigating: https://discord.com/channels/595317990191398933/992060581832032316/1357236057569099849

Ggruntlord6 I haven't used this but if it comes up for me and they still don't have the dyna...

Vlad Lykhonis•5/9/25, 9:23 PM

I am using workers for platforms, so allocating new D1 and binding it to a my clients scripts should be no problem. I was curious if there is some more general approach for sharding, but indeed seems more situational aka application specifics

gruntlord6•5/9/25, 9:24 PM

yea its always going to come down to what data is suitable to be sharded. You can approach it all kinds of ways. Does each client get a db? each organization? each member of the org? etc

gruntlord6•5/9/25, 9:25 PM

in theory you could have a central db that just maps data from other dbs, that would probably work for most data sets but be more complicated to manage

Ggruntlord6 in theory you could have a central db that just maps data from other dbs, that w...

lambrospetrou•5/10/25, 8:21 AM

Many folks do this, but I recommend against it. If you put a single database (or even Durable Object) in the hot path of every request you lost all the benefits of sharding across multiple of them. If that one DB/DO is down, then you halt.
It's totally OK to have one control plane database for things like managing your resources during creation/deletion/listing, but for the hot path and data plane operations of your application you should not depend on that single DB/DO.
I wrote about this pattern here: https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/

Cloudflare Docs

Control and data plane architectural pattern for Durable Objects

Durable Objects are built on-top of Cloudflare Workers spanning several locations across our global infrastructure network.
Each Durable Object instance has its own durable storage persisted across requests, in-memory state, single-threaded execution, and can be placed in a specific region.

will smith•5/11/25, 2:42 PM

is it me or is database editing in the UI broken?
It's straight up editing the wrong row. Video attached
https://dash.cloudflare.com/a3f6e70849e6eb5ca1ebaf6083bc84ba/workers/d1/databases/79a7d7de-4213-4a3a-b53c-e9ac3c1eda91/tables/Page?page=1&perPage=10

Wwill smith is it me or is database editing in the UI broken? It's straight up editing the w...

Brayden•5/11/25, 3:43 PM

If you go back to the D1 resource list page and click the “…” menu, then “Explore data”, try using this new experience & let me know if it works.

BBrayden If you go back to the D1 resource list page and click the “…” menu, then “Explor...

will smith•5/11/25, 4:05 PM

yep this one's working thanks

BBrayden If you go back to the D1 resource list page and click the “…” menu, then “Explor...

gruntlord6•5/11/25, 10:46 PM

that looks a lot nicer, any chance it will have sortable columns like the other version does?

Ggruntlord6 that looks a lot nicer, any chance it will have sortable columns like the other ...

Brayden•5/11/25, 11:54 PM

This week, stay tuned

Llambrospetrou I can definitely argue that any latency comparison between REST API and Worker B...

mon•5/12/25, 1:43 AM

Here's some typical numbers for small and large returns, where I take a

performance.now()

performance.now()

before and after the call to

bind(...).all()

bind(...).all()

to identify the total time.

row fetch took 206 ms
result serialises to 19906 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 0.4254
  },
  "duration": 0.4254,
  "changes": 0,
  "last_row_id": 705,
  "changed_db": false,
  "size_after": 10797056,
  "rows_read": 2,
  "rows_written": 0
}

row fetch took 3748 ms
result serialises to 5100116 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 2.4634
  },
  "duration": 2.4634,
  "changes": 0,
  "last_row_id": 840,
  "changed_db": false,
  "size_after": 13021184,
  "rows_read": 100,
  "rows_written": 0
}

row fetch took 206 ms
result serialises to 19906 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 0.4254
  },
  "duration": 0.4254,
  "changes": 0,
  "last_row_id": 705,
  "changed_db": false,
  "size_after": 10797056,
  "rows_read": 2,
  "rows_written": 0
}

row fetch took 3748 ms
result serialises to 5100116 bytes of JSON
result meta {
  "served_by": "v3-prod",
  "served_by_region": "APAC",
  "served_by_primary": true,
  "timings": {
    "sql_duration_ms": 2.4634
  },
  "duration": 2.4634,
  "changes": 0,
  "last_row_id": 840,
  "changed_db": false,
  "size_after": 13021184,
  "rows_read": 100,
  "rows_written": 0
}

Mmon Here's some typical numbers for small and large returns, where I take a `perform...

lambrospetrou•5/12/25, 6:33 AM

Your worker doing this is in APAC too? Judging from the fast response my guess would be that even if it is, probably far away from the DB still.

Llambrospetrou Your worker doing this is in APAC too? Judging from the fast response my guess w...

mon•5/12/25, 6:36 AM

I'm in OC so I assume the worker will be too - similar timings for deployed worker vs

wrangler dev --remote

wrangler dev --remote

mon•5/12/25, 6:36 AM

No smart placement

Mmon I'm in OC so I assume the worker will be too - similar timings for deployed work...

lambrospetrou•5/12/25, 6:38 AM

OK OC to APAC is super varied unfortunately. Without digging into it, my guess is that the 5MB response size hits the limitation we know about the JSRPC protocol (not D1 specific), and gets worse when over OC->APAC. I will raise it again with the team since it's been in discussion for some time.

mon•5/12/25, 6:39 AM

Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably try and migrate this code to Durable Objects to avoid the cross-region shenanigans - I've been told we tried and failed before, but with SQLite in DO now I think it'll be smooth sailing.

mon•5/12/25, 6:43 AM

Well, not perfectly smooth sailing because

[WARNING] SQLite in Durable Objects is only supported in local mode.

[WARNING] SQLite in Durable Objects is only supported in local mode.

, but I can probably work around that for the time being

Mmon Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably...

lambrospetrou•5/12/25, 6:45 AM

BTW, the limit I mentioned applies to DOs as well, but at least there your code is near the database so you can do your main logic there to reduce the 5MB result further down before returning it out of the DO.

mon•5/12/25, 6:46 AM

Yep, and that's exactly what my plan is - the rows are all read, lightly processed and saved to R2. By moving this work into the DO, we entirely eliminate the RPC overhead.

mon•5/12/25, 6:47 AM

Plus it's information about a user session, so I get to simplify all the queries and just have one DO per session instead of filtering the rows.

mon•5/12/25, 6:47 AM

Honestly it really is the perfect usecase for DOs and I'm sad it didn't work out the first time

ByteBeast•5/12/25, 11:10 AM

I have one table in which I have auth column when i query it into worker and fetch value in that column it gets converted in to **. what is this and how can I get actual value?

yinxingmaiming6409•5/13/25, 8:55 AM

I want to experience RealtimeKit. Can someone review it for me?

Yyinxingmaiming6409 I want to experience RealtimeKit. Can someone review it for me?

1984 Ford LaserOP•5/13/25, 11:43 AM

You need to be posting in #realtime

kchro3•5/13/25, 4:29 PM

Hey, I started seeing a bunch of

D1_ERROR: Network connection lost

D1_ERROR: Network connection lost

(even after adding retries) where every write would fail.

the trace looked like this:

" at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1DatabaseSessionAlwaysPrimary.batch (cloudflare-internal:d1-api:80:23) at async Object.queue (index.js:6144:5)",

" at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1DatabaseSessionAlwaysPrimary.batch (cloudflare-internal:d1-api:80:23) at async Object.queue (index.js:6144:5)",

i was trying to use batch on a list of insert statements like this:

await env.D1.batch(
  batch.map((item) => stmt.bind(...))
);

await env.D1.batch(
  batch.map((item) => stmt.bind(...))
);

is there a soft limit on how large a batch can be? i was able to resolve it by breaking up the batches into smaller batches of 25, but it wasn't causing problems before and the full batch size is <100

Tuna•5/13/25, 6:59 PM

Hi,

I have workers, durableObject lets say one durableObject is for keep track of uploads from client to server (i have my own API for multipart upload), the second is for access to R2 and also used as life cycle, alarm is set for TTL and after that its deleting the file/s from R2.

Im wondering what will be best practice, i want the durableObject to sync R1, like when user upload files, it will keep track in durable object but also sync the global database R1 which will be per user, and i will need one database for all users.
i also have an api gateway worker with itty router to handle rest API.

should i update the D1 from the router handler / workers ? should i clean the R2 from the alarm?
or should i use queues?
if i use queues for async operations, they need to be bounded to the same worker or i can set the main worker to be producer and create a second worker to be the consumer ?

parisholley•5/13/25, 7:22 PM

anyone seeing a higher than usual amount of D1 errors (timeout, connection lost, etc)

row fetch took 206 ms result serialises to 19906 bytes of JSON result meta { "served_by": "v3-prod", "served_by_region": "APAC", "served_by_primary": true, "timings": { "sql_duration_ms": 0.4254 }, "duration": 0.4254, "changes": 0, "last_row_id": 705, "changed_db": false, "size_after": 10797056, "rows_read": 2, "rows_written": 0 } row fetch took 3748 ms result serialises to 5100116 bytes of JSON result meta { "served_by": "v3-prod", "served_by_region": "APAC", "served_by_primary": true, "timings": { "sql_duration_ms": 2.4634 }, "duration": 2.4634, "changes": 0, "last_row_id": 840, "changed_db": false, "size_after": 13021184, "rows_read": 100, "rows_written": 0 }

" at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1DatabaseSessionAlwaysPrimary.batch (cloudflare-internal:d1-api:80:23) at async Object.queue (index.js:6144:5)",

Durable Objects

Similar Threads

Similar Threads

Similar Threads