`Network connection lost.` are transient errors reaching the location/datacenter of your database.

Network connection lost.

Network connection lost.

Network connection lost.

Network connection lost. are transient errors reaching the location/datacenter of your database.

My suggestion is to add retries for some common errors. See an example in the D1 read replication demo of how to do retries: https://github.com/cloudflare/templates/blob/main/d1-starter-sessions-api-template/src/index.ts#L108
And how it's used: https://github.com/cloudflare/templates/blob/c624b31b63568adebcba1b63f600dbc81cf3a5fa/d1-starter-sessions-api-template/src/index.ts#L59

Llambrospetrou `Network connection lost.` are transient errors reaching the location/datacenter...

ajgeiss0702•4/11/25, 12:18 AM

I've gotten this error before and did retries there, but after enabling replication the error is happening way more than it did before (even though the dashboard shows that no replicas have been made yet)

ajgeiss0702•4/11/25, 12:18 AM

before, this error would only happen on some of my hottest paths on the most popular day of the week, but now its even happening today (which is a low day) on paths that don't get a ton of traffic

Aajgeiss0702 I've gotten this error before and did retries there, but after enabling replicat...

Zack•4/11/25, 12:18 AM

Have you moved your worker to use the sessions API?

ZZack Have you moved your worker to use the sessions API?

ajgeiss0702•4/11/25, 12:19 AM

at first I didnt, but now I have and it still happens occassionally.
When I first deployed the sessions api for one of my DBs, all requests failed with this error for a few minutes

Zack•4/11/25, 12:19 AM

What bookmark constraint are you using?

Aajgeiss0702 before, this error would only happen on some of my hottest paths on the most pop...

lambrospetrouOP•4/11/25, 12:20 AM

What's a "low day"? These errors are not dependent on your workload, they depend on the health of the datacenter your worker runs and of the database.

ZZack Have you moved your worker to use the sessions API?

lambrospetrouOP•4/11/25, 12:20 AM

That's irrelevant

ZZack What bookmark constraint are you using?

ajgeiss0702•4/11/25, 12:20 AM

im not specifying (which should default to first-unconstrainedfirst-unconstrained) just doing withSession()withSession()

Llambrospetrou That's irrelevant

Zack•4/11/25, 12:22 AM

Surely not if the query is always using 'firstPrimary' without reusing the session?

Zack•4/11/25, 12:22 AM

As then it will only ever hit the primary instance

Zack•4/11/25, 12:22 AM

Still shouldn't be seeing those errors presumably, but it'd explain why they only have a single instance count

ZZack Surely not if the query is always using 'firstPrimary' without reusing the sessi...

lambrospetrouOP•4/11/25, 12:23 AM

The discussion is about the network connection lost errors, not about what database instance is hit

Llambrospetrou What's a "low day"? These errors are not dependent on your workload, they depend...

ajgeiss0702•4/11/25, 12:24 AM

the site im testing this with is a site centered around an event that happens every friday evening, so any day other than friday has very little traffic compared to friday.
The "most popular" page that ive seen with this error has only had 45 requests in the last 24h, of which 6 have failed with this error.
there are other paths that have thrown this error, but thats the one with the most requests in the past 24h

ajgeiss0702•4/11/25, 12:27 AM

before enabling replicas this morning, there were 0 errors from any of these less popular pages. Now theres a few dozen

Zack•4/11/25, 12:40 AM

Alright, so after the success of using withSession('first-unconstrained')withSession('first-unconstrained') in dev, I decided to try and push that to prod. Now the database is entirely unresponsive and hasn't managed to serve a successful request for the last 10 minutes. Continually getting network connection lost errors

ZZack Alright, so after the success of using `withSession('first-unconstrained')` in d...

ajgeiss0702•4/11/25, 12:41 AM

that happened to me for one of my DBs too. It started working after a few minutes

Zack•4/11/25, 12:42 AM

I'm giving it a bit of time to recover from the sudden load, but can't really give it much longer, it's been ~15 minutes now

ajgeiss0702•4/11/25, 12:43 AM

looks like mine took about 17 minutes to recover

Zack•4/11/25, 12:43 AM

I'll give it another 5, then I will have to revert as we are serving entirely from cache atm, any misses are throwing 500's

Zack•4/11/25, 12:50 AM

Alright, rolling back. Database never recovered

ajgeiss0702•4/11/25, 2:38 AM

When using the sessions api with a database that does not have replication enabled yet, sessions are just ignored and all queries are passed to the master, right?

Chaika•4/11/25, 2:52 AM

It's documented that sessions api works w/o replication:

Sessions API works with databases that do not have read replication enabled, so it is safe to run code with Sessions API even after disabling read replication.

https://developers.cloudflare.com/d1/best-practices/read-replication/#benefits-of-read-replication
and first-unconstrained just means any available, so should just work, haven't tested it myself though

ajgeiss0702•4/11/25, 2:54 AM

makes sense, thanks

ajgeiss0702•4/11/25, 2:54 AM

it definitely works in local dev, so it probably works in prod too

ajgeiss0702•4/11/25, 3:17 AM

yep can confirm, using sessions appears to work fine on a db that doesn't have replication enabled

ajgeiss0702•4/11/25, 3:17 AM

I'm only calling withSession()withSession() though, i'm not trying to use bookmarks on DBs that dont have replication enabled, so i cant confirm if that works or not. I imagine it would though

Whitey•4/11/25, 3:27 AM

I'm playing around with CF pages and D1 for a hobby project. Drizzle handles my database migrations and have I the migrations directory set in

wrangler.jsonc

wrangler.jsonc

. I can successfully run the migrations if I do wrangler d1 migrations apply <database>wrangler d1 migrations apply <database>, but when I then do wrangler pages dev --d1 MY_DB=mydbwrangler pages dev --d1 MY_DB=mydb to run my app locally, drizzle complains about a missing table, seemingly my migrations haven't been run after all.

I also noticed after running both commands there are two .sqlite.sqlite files in .wrangler/state/v3/d1/miniflare-D1DatabaseObject.wrangler/state/v3/d1/miniflare-D1DatabaseObject

does the migrations applymigrations apply command operate on a different local database than wrangler pages devwrangler pages dev? Just trying to figure out the best way to set this up, and feel like I'm using wrangler incorrectly here

Aajgeiss0702 I'm only calling `withSession()` though, i'm not trying to use bookmarks on DBs ...

lambrospetrouOP•4/11/25, 7:26 AM

Even with using a bookmark, when you turn off replication and eventually all replicas are killed, your queries will go to the primary. The primary has the most up-to-date database since all writes go there so it will always be caught up with the provided bookmark instantly.

Llambrospetrou Even with using a bookmark, when you turn off replication and eventually all rep...

lambrospetrouOP•4/11/25, 7:32 AM

The way we thought about it is that since the D1DatabaseSessionD1DatabaseSession has the same prepare/batch()prepare/batch() methods as the top level D1 Binding your code doing D1 queries should just accept D1DatabaseSessionD1DatabaseSession. And therefore you can pass in your code either the top level binding or the session create by withSession()withSession().

So, switching between always using the top level binding or the session is a single line change. Like I did in https://github.com/cloudflare/templates/blob/c624b31b63568adebcba1b63f600dbc81cf3a5fa/d1-starter-sessions-api-template/src/index.ts#L52

Zack•4/11/25, 9:09 AM

Is someone able to check through the d1 issues I experienced yesterday and see why my database never became healthy after switching to the replicated database instead of 6 individually placed databases?

Zack•4/11/25, 9:14 AM

These were all the errors we experienced from D1, which was most of our cache missed requests. I think a few requests successfully ran, but it was a very low percentage and the database fell over immediately afterwards again

Zack•4/11/25, 10:08 AM

More on the above, tried to slowly roll in the load on the replicated database, which all seemed fine until hitting 75% traffic, which has now caused p80 to rocket and errors to start appearing. Waiting to see if it dies down.

Zack•4/11/25, 10:09 AM

Can someone at cloudflare confirm a suspicion I have? Non-replicated D1 feels like either, the instances (DOs) live for longer before being shutdown or are just faster at starting up? Some of the problem seems to be that a region gets hit where the instance isn't available currently and that causes slowdowns while that replica comes online

ZZack These were all the errors we experienced from D1, which was most of our cache mi...

lambrospetrouOP•4/11/25, 10:28 AM

The team will be looking your database's errors today.

ZZack Can someone at cloudflare confirm a suspicion I have? Non-replicated D1 feels li...

lambrospetrouOP•4/11/25, 10:29 AM

Non-replicated D1 feels like either, the instances (DOs) live for longer before being shutdown or are just faster at starting up?

No. Replicas start up in the same way as any other database, and non-replicated databases do not live longer.

Llambrospetrou > Non-replicated D1 feels like either, the instances (DOs) live for longer befor...

Zack•4/11/25, 10:29 AM

Interesting, this behaviour is pretty confusing then

Llambrospetrou The team will be looking your database's errors today.

Zack•4/11/25, 10:29 AM

Thanks, let me know if there's anything you need from me

ZZack Interesting, this behaviour is pretty confusing then

lambrospetrouOP•4/11/25, 10:30 AM

Some of the problem seems to be that a region gets hit where the instance isn't available currently and that causes slowdowns while that replica comes online

This is true. If you just hit your replication-enabled database from a region for the first time, the replica there will have to start up.

Zack•4/11/25, 10:30 AM

Is it more useful if there is load on the database when they go to look at it, or will they be able to just look at logs and metrics from now and when I tried also yesterday

Llambrospetrou > Some of the problem seems to be that a region gets hit where the instance isn'...

Zack•4/11/25, 10:30 AM

Right, the confusing part to me is that we saw none of this behaviour when running 6 seperate D1 instances located within each region

ZZack Right, the confusing part to me is that we saw none of this behaviour when runni...

lambrospetrouOP•4/11/25, 10:31 AM

I doubt that, since your normal database would have to start up as well when first queried.

Zack•4/11/25, 10:31 AM

Also, this database has zero writes occurring, it's solely just reads

Llambrospetrou I doubt that, since your normal database would have to start up as well when fir...

Zack•4/11/25, 10:31 AM

I have metrics to prove it

Zack•4/11/25, 10:31 AM

It may not be the startups causing the problem, but we didn't have these problems when running 6 seperate instances

Zack•4/11/25, 10:32 AM

Lets see what the team find when they take a look into it

ZZack It may not be the startups causing the problem, but we didn't have _these_ probl...

lambrospetrouOP•4/11/25, 10:33 AM

When did you first enable replication for your database?

Zack•4/11/25, 10:34 AM

We have 7 databases, all with the same data. 6 to have one for each region, and the 7th in preperation for read replicas. I turned read replicas on yesterday after the release, and toggled it a couple times to see if that fixed the behaviour.

Zack•4/11/25, 10:34 AM

This instance primary is based in WEUR

Zack•4/11/25, 10:35 AM

This also gives us the ability to switch between the two setups easily with just a worker rollback

Zack•4/11/25, 10:38 AM

But yea, just reverted back to our old setup now, and immediately we return to no errors and latencies return back to normal ranges. Not a single request >2 seconds

`Network connection lost.` are transient errors reaching the location/datacenter of your database.

Similar Threads

Similar Threads

Similar Threads