Cloudflare Developers•8mo ago

Actually, I think the error might be something else

MMirith Actually, I think the error might be something else

gruntlord6•5/4/25, 5:40 PM

the worker is transmitting the d1 operation by web socket?

MirithOP•5/4/25, 6:01 PM

No. The problem involves the websocket but the operation is just not getting called

MMirith Is there a way to just look at the console.log output remotely?

Chaika•5/4/25, 6:34 PM

wrangler tail https://developers.cloudflare.com/workers/observability/logs/real-time-logs/ but you'll only see logs when the websocket ends

Ggruntlord6 batch is transactional https://developers.cloudflare.com/d1/worker-api/d1-databa...

Successful-Arm-3762•5/4/25, 7:03 PM

ooh, thanks for letting me know! this is helpful! I thought batch just reduces round trips, good to know!

gruntlord6•5/4/25, 7:04 PM

I think we all get caught by that one, luckily the docs are there

chook•5/6/25, 1:36 AM

Has anyone encountered a 504 response when running large migrations via wrangler?

I know there is this issue

If so how did you get around the error? I've tried running

d1 execute --file

d1 execute --file

with the same result

GitHub

🐛 BUG: `504 Gateway Time-out` when applying migrations in D1 · ...

Which Cloudflare product(s) does this pertain to? D1 What versions are you using? 3.102.0 [Wrangler], 20.18.0 [Node.js] What operating system and version are you using? Mac Sequoia 15.2 Please prov...

Cchook Has anyone encountered a 504 response when running large migrations via wrangler...

lambrospetrou•5/6/25, 8:47 AM

How many rows and megabytes are you importing? Have you tried chunking the file into smaller imports? We still didn't get around to optimizing the import flows, and it has several limitations for long running jobs.

daycompacted•5/6/25, 5:25 PM

In order to avoid the 1m lag with KV propagating to all regions, I'm thinking to migrate everything into D1. It seems as though D1 is also cheaper for read/writes/storage. Aside from the difference in performance, is there any reason to stick with KV?

Ddaycompacted In order to avoid the 1m lag with KV propagating to all regions, I'm thinking to...

lambrospetrou•5/6/25, 5:29 PM

Scaling and throughput properties of KV and D1 are wildly different. D1 is a single Durable Object in a single location. KV is cached globally after first access.
Very highly dependent on your workload, requests per second, reads vs writes, locations you access it from, cachability, how much data, etc.

mon•5/7/25, 4:40 AM

Sanity check: wrangler no longer has the

--experimental-backend

--experimental-backend

flag, and

wrangler d1 info

wrangler d1 info

no longer returns the version. Have all beta DBs been silently upgraded to the new backend?
Just trying to remove it as an explanation for some poor performance.

nejc•5/7/25, 5:15 AM

hi all, not sure if this is the correct channel, but would it be possible to add filter for max_databaseSizeBytes https://api.cloudflare.com/client/v4/graphql for d1StorageAdaptiveGroups?

Mmon Sanity check: wrangler no longer has the `--experimental-backend` flag, and `wra...

lambrospetrou•5/7/25, 6:02 AM

Do you mean "alpha" databases? We did not upgrade alpha databases to the production version, and no plans to do so. Alpha databases are deprecated and should be rejecting queries.
I will report the missing

version

version

info and will fix.

Llambrospetrou Do you mean "alpha" databases? We did not upgrade alpha databases to the product...

mon•5/7/25, 6:10 AM

Excellent, thanks!

Llambrospetrou Do you mean "alpha" databases? We did not upgrade alpha databases to the product...

lambrospetrou•5/7/25, 6:24 AM

I just checked the wrangler code, and the

version

version

is shown in

wrangler d1 info

wrangler d1 info

only if it's an alpha database. So, since you are not seeing it your database is not alpha. If you send me the database ID (DM) I can check its version and confirm to you.

Llambrospetrou I just checked the wrangler code, and the `version` is shown in `wrangler d1 inf...

mon•5/7/25, 6:28 AM

I'm happy with that conclusion - I'll try other performance avenues!

mon•5/8/25, 12:59 AM

Another question regarding read replicas, mainly because I don't quite understand CF regions - is there only ever one replica per region, or is it more local than that?
Eg: APAC is a pretty large region, if someone from Perth, Australia makes a request would the replica be made there or is it a more central location like Sydney? If a second request comes from Tokyo, would it use the first replica or make a new one nearby?

Mmon Another question regarding read replicas, mainly because I don't quite understan...

Isaac McFadyen•5/8/25, 1:52 AM

Per region. And replicas are created once when you enable them, they won't be created on-demand like you describe in your second scenario.

Isaac McFadyen•5/8/25, 1:52 AM

I.e requests from APAC will go to the single replica in APAC (or whatever the replication regioning system is, I'm not sure it's as clearly defined as that), which is created when you enable replicas initially.

mon•5/8/25, 2:08 AM

Gotcha, thanks

Mmon Another question regarding read replicas, mainly because I don't quite understan...

Chaika•5/8/25, 2:31 AM

Apac is large but Cloudflare subdivides it. Australia is part of their Oceanic region https://where.durableobjects.live/region/oc and Tokyo would be part of Asia Pacific https://where.durableobjects.live/region/apac
two different read replica locations

Chaika•5/8/25, 2:33 AM

Still what you described can essentially happen with Singapore vs Tokyo for example, with same region. Could have your read replica placed in Singapore even if you only get traffic from Japan. Asia Pacific is def the messiest region

mon•5/8/25, 2:35 AM

Hmm, when using the "served from" metadata in a response, will it be precise enough to know if it was Sydney vs Tokyo, or will it just return APAC?

Mmon Hmm, when using the "served from" metadata in a response, will it be precise eno...

Chaika•5/8/25, 2:35 AM

just apac

Chaika•5/8/25, 2:36 AM

(Sydney would return OC not APAC)

Chaika•5/8/25, 2:36 AM

Nothing you can do if it's at a suboptimal location within a region eitherway tho

mon•5/8/25, 2:37 AM

Right right, shame there's no visibility there, though I imagine you do that and people will start asking for precise location hints

Chaika•5/8/25, 2:38 AM

yea, something I don't think CF would ever do as it'd limit their ability to remove locations within a region/more dependent on specific locations
think the more likely solution is just more smart placement within the region

mon•5/8/25, 3:11 AM

Do you know if the REST api can "leak" the location just by the IP it gets routed to? I can probably just test that myself now...
I'm thinking that smart placement + read replicas are going to be (mostly) a panacea to my performance problems

mon•5/8/25, 3:12 AM

I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs) is about 2.5 seconds worse, and I'm curious if you're seralising to JSON using a JS thing behind the scenes, which would explain that slowness.

Mmon Do you know if the REST api can "leak" the location just by the IP it gets route...

Chaika•5/8/25, 3:14 AM

hmm by the ip it gets routed to? The Cloudflare Rest API is just behind the normal proxy, and I believe it still sends all queries to one of the cores first (eu, us) making apac queries via REST rather slow

mon•5/8/25, 3:14 AM

My actual use case is via workers, I'm just making sure it's not my worker code by testing with REST, and seeing very similar timings

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

Chaika•5/8/25, 3:22 AM

Some of its part of the open source runtime: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L159
I believe it's something like d1 worker -> d1 durable object, and Workers RPC calls (which uses V8 Serialization for values and Cap'n proto for the protocol itself)

Chaika•5/8/25, 3:23 AM

I have noticed the same slowness myself with returning lots of data though

Chaika•5/8/25, 3:26 AM

hmm perhaps there's more json then I thought: https://github.com/cloudflare/workerd/blob/7bfa941f918f7a43c5d28cda948d09fdc9400c17/src/cloudflare/internal/d1-api.ts#L239C1-L239C33
interesting. They mentioned publicly about switching to RPC before, but maybe just the outer layer

mon•5/8/25, 3:27 AM

That would certainly explain it, it seems to match with the JSON serialisation benchmarks I've seen for v8

Mmon Another question regarding read replicas, mainly because I don't quite understan...

lambrospetrou•5/8/25, 9:18 AM

For now yes, that's correct, one static replica per region. There is certainly a roadmap to extend that and improve it over the next several months to be more dynamic and follow your traffic patterns, but it's still some time away.

Vlad Lykhonis•5/8/25, 8:53 PM

Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size 10GB what are strategies to mitigate and continue to scale large databases?

I’m leaning towards some sharding technique based on ids ranges of some sort. I have not looked into this deeply yet, why I’m curious to hear if anyone already see some more general reliable way to scale above 10Gb.

VVlad Lykhonis Hi. I’m sure this has been asked before. When it comes to hard limit of D1 size ...

1984 Ford Laser•5/8/25, 10:05 PM

Durable Objects

Mmon I also note that TTFB for rows with some amount of data (100 rows of 16kb BLOBs)...

mon•5/9/25, 1:29 AM

So I did some actual benchmarks and this doesn't make sense really (serialisation should take tens of ms). I went back in the discord history, and every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.
Is there any knowledge/work behind the scenes on this? I'd be perfectly happy with "it's [a network layer | unoptimised code | something else] and we know about it" just to get some confidence. This is APAC, either REST or using the Workers SDK, warm DB (seconds between requests) with no concurrent writes during the read, same performance characteristics through both services.

11984 Ford Laser Durable Objects

Vlad Lykhonis•5/9/25, 1:40 AM

What durable objects? How is it helpful.

VVlad Lykhonis What durable objects? How is it helpful.

Isaac McFadyen•5/9/25, 1:45 AM

Each DO can have it's own SQLite database, and you can spawn unlimited number of DOs (within your account-wide limit): https://blog.cloudflare.com/sqlite-in-durable-objects/

Isaac McFadyen•5/9/25, 1:45 AM

https://developers.cloudflare.com/durable-objects/#what-are-durable-objects

IIsaac McFadyen Each DO can have it's own SQLite database, and you can spawn unlimited number of...

Vlad Lykhonis•5/9/25, 5:05 AM

That wasn’t a question. I can create 50k D1s too and extend limit. Question was about what to do once D1 reaches 10GB storage.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

Isaac McFadyen•5/9/25, 5:06 AM

If you create multiple D1 DBs you have to bind them manually to a Worker. A DO can be created and invoked without manually redeploying the Worker.

Isaac McFadyen•5/9/25, 5:07 AM

As for sharding or similar strategies, that's entirely application-specific.

IIsaac McFadyen As for sharding or similar strategies, that's entirely application-specific.

Vlad Lykhonis•5/9/25, 5:17 AM

This was the question.

VVlad Lykhonis That wasn’t a question. I can create 50k D1s too and extend limit. Question was ...

lambrospetrou•5/9/25, 7:43 AM

The 10GB is a hard limit so you need to either migrate your data out of the single database if you get there, or plan ahead by using multiple databases or something else like sharding across Durable Objects or Hyperdrive+traditional database.

Mmon So I did some _actual_ benchmarks and this doesn't make sense really (serialisat...

lambrospetrou•5/9/25, 7:46 AM

If you make a clear question I can answer it for you. Your previous question about JSON serialization was answered already linking to the code of the D1 binding which yes, it does JSON parsing for the response from the D1 service.

Our D1 worker doesn't do any kind of intermediate serialization/parsing at all, but the result is JSON serialized from the underlying SQLite before being send out. So, there is a JSON serialization once, and a JSON deserialization inside your worker for the result. Nothing special in that serialization, you will get the same perf as

JSON.parse/JSON.stringify

JSON.parse/JSON.stringify

every time someone has complained about D1 latency, it's either a temporary spike for everyone, or never responded to by a cloudflare employee.

Be concrete please. "Latency" is not a single thing. There are millions of reasons why something can be slow. User error, network errors, Cloudflare bugs, and lots of other things.

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

lambrospetrou•5/9/25, 7:50 AM

There is a certain bug we know internally (not specific to D1 but to JSRPC) that could lead to slow network transfers if it's several MBs, but at least from your previous comment it seems that you only respond 1-2MBs?

Llambrospetrou If you make a clear question I can answer it for you. Your previous question abo...

mon•5/9/25, 7:59 AM

Sorry for muddying the waters with the JSON serialisation, I was stupid and took benchmark data from (what turned out to be) quite an old version of V8, instead of just testing it myself to begin with.

In my case, when I say latency I really mean TTFB of the response. Based on the posts I've seen and my own tests, it's a reasonably common note that larger (where I've seen that described as "10,000 rows" and "some megabytes") queries, while executing within milliseconds, will take multiple seconds extra time to start being delivered to the recipient.
Concretely, when I fetch 100 rows of 16KiB BLOBs, if I do SELECT data ...SELECT data ... the response takes ~3500ms to start arriving. RunningSELECT COUNT(data)SELECT COUNT(data) as a control takes about 700ms.
This is when using the REST API via the Cloudflare dashboard, and looking at Firefox's network timings.
I see similar times in a Worker, which should have clued me in that it wasn't the JSON to begin with. The worker and I are both in OC region, the database in APAC.

Actually, I think the error might be something else

Similar Threads