Cloudflare Developers•9mo ago

Unless there is an incident the normal replication lag is a round trip from the primary to the repli

Unless there is an incident the normal replication lag is a round trip from the primary to the replicas, yes.

Llambrospetrou Unless there is an incident the normal replication lag is a round trip from the ...

ajgeiss0702•4/10/25, 8:21 PM

cool, thank you

Samantas5855•4/10/25, 8:22 PM

Ive been facing issues, not all my api pushes make it to the database

Samantas5855•4/10/25, 8:22 PM

I assume Im not the only one

Zack•4/10/25, 8:23 PM

IIsaac McFadyen D1 binding issue is being looked at by engineering internally, thanks for the re...

ajgeiss0702•4/10/25, 8:24 PM

Zack•4/10/25, 8:24 PM

Getting this on just the replicated database that I am testing in dev, not even switched to the session syntax yet

Isaac McFadyen•4/10/25, 8:24 PM

See my message above (if Discord is working enough for it to come in, haha - they're also having an incident)

Zack•4/10/25, 8:25 PM

Apologies, I thought that was with reference to people having trouble spawning workers with D1 bindings, missed the other messages with similar errors

ZZack Apologies, I thought that was with reference to people having trouble spawning w...

Isaac McFadyen•4/10/25, 8:26 PM

No worries - yeah I don't know specifics of what would be affected, just that some internal API is having issues, R2+D1 are affected, and that engineering is engaged internally

Zack•4/10/25, 8:26 PM

Fair enough, my main addition then is that this isn't affecting our production databases, just this one with replication enabled. Unsure if that's relevant or just coincidence

Nikolas•4/10/25, 8:27 PM

The problem we currently face is, when the d1 write succeeds and doesn't throw "internal error", it doesn't write all the columns, only the first ones. We do a big insert query through the API. I assume there is a grater issue with d1 right now?

NNikolas The problem we currently face is, when the d1 write succeeds and doesn't throw "...

Isaac McFadyen•4/10/25, 8:28 PM

There is but I'm not sure if that would be a result of the issues or not, sorry.

IIsaac McFadyen D1 binding issue is being looked at by engineering internally, thanks for the re...

datacenterdude•4/10/25, 8:45 PM

Appears to be back online! Appreciate the quick response to the team!

Isaac McFadyen•4/10/25, 8:45 PM

Yup, issues should be resolved! Thanks again for reporting

IIsaac McFadyen Yup, issues should be resolved! Thanks again for reporting 😄

Zack•4/10/25, 8:55 PM

Still seeing issues, though different

Zack•4/10/25, 8:57 PM

Correction, still seeing the same errors from before:

ZZack Still seeing issues, though different

Chaika•4/10/25, 9:30 PM

how frequent?

CChaika how frequent?

Zack•4/10/25, 9:32 PM

Every request when I was testing

Chaika•4/10/25, 9:33 PM

do you have replication enabled on the db? I see some on my replicated db, but none w/o

Zack•4/10/25, 9:33 PM

I can test again shortly

CChaika do you have replication enabled on the db? I see some on my replicated db, but n...

Zack•4/10/25, 9:33 PM

Yes, replicated, non replicated databases are totally error free

Zack•4/10/25, 9:33 PM

FWIW this isn't a big issue for me, but if it's more widespread I am happy to help with providing errors and retesting

Zack•4/10/25, 9:33 PM

I am just testing the replicated d1 in dev right now but it just won't work for any requests

Samantas5855•4/10/25, 9:54 PM

apparently my issues seem to be due to caching in pages functions

ZZack I am just testing the replicated d1 in dev right now but it just won't work for ...

Milan•4/10/25, 10:36 PM

Dumb question but wdym "in dev"?

MMilan Dumb question but wdym "in dev"?

Zack•4/10/25, 10:37 PM

As in, our dev environment

Zack•4/10/25, 10:37 PM

I've not pushed using replication into our production environment yet, so it's not a big issue for me

Zack•4/10/25, 10:37 PM

I am still experiencing the problems though

Milan•4/10/25, 10:37 PM

Like, running wrangler --dev remotewrangler --dev remote, or another namespace separate from your production namespace, but still deployed to cloudflares network?

Zack•4/10/25, 10:38 PM

All deployed to cloudflare, we deploy via terraform

Zack•4/10/25, 10:38 PM

Seperate database to production

Zack•4/10/25, 10:38 PM

It's "production" as far as you guys know

Milan•4/10/25, 10:41 PM

Thanks for clarifying!

MMilan Thanks for clarifying!

Zack•4/10/25, 10:46 PM

Some more info, I disable read replication on the database, redeployed the worker, everything worked fine. Enabled replication again, redeployed the worker, constant errors again

Zack•4/10/25, 10:46 PM

D1 ID: c60521e7-59bd-4186-81e7-9a676edf9b75
Incase you wanted to look into it. If you need the worker ID, just lmk what specifically you need

Aajgeiss0702 db id is `221100ba-5ab9-4cda-8cfa-69c17ea546ab` if it helps (account id is `f55b...

ajgeiss0702•4/10/25, 10:59 PM

i am also getting the issue occassionally

MMilan Thanks for clarifying!

Zack•4/10/25, 11:04 PM

Sorry to reply one more time, but I just wanted to say that I hadn't moved to use withSessionwithSession yet, might just be coincidence, but I just deployed the worker with that change, and now the replicated D1 database is working just fine

ZZack Sorry to reply one more time, but I just wanted to say that I hadn't moved to us...

Milan•4/10/25, 11:22 PM

I'm not familiar enough with D1's API to know whether this would cause problems or not, but can get back to you tomorrow

Zack•4/10/25, 11:23 PM

My understanding is that without using the session API it will only ever route to the primary instance of D1

Zack•4/10/25, 11:23 PM

But it shouldn't just fall over and constantly fail

Milan•4/10/25, 11:26 PM

I think that's probably right, it's possible the redeploy moved your primary somewhere else and the previous host was just having some weird ephemeral issue, but we'll figure it out and let you know

Zack•4/10/25, 11:27 PM

Sounds good

ZZack My understanding is that without using the session API it will only ever route t...

lambrospetrouOP•4/10/25, 11:32 PM

Without using the Sessions API (withSession()withSession()) all requests are sent to the primary explicitly. Under the hood the D1 binding has already been using the Sessions API for non replicated databases for months now, and it sends first-primaryfirst-primary when you don't use withSession()withSession() yourself.

Zack•4/10/25, 11:33 PM

Makes sense, to be more specific then, I now use withSession('first-unconstrained')withSession('first-unconstrained') which is working as I would expect, and not experiencing those same errors

ajgeiss0702•4/10/25, 11:56 PM

I switched to using the sessions api for one of my D1 DBs (that has replication enabled) and now almost every request is failing with the

D1_ERROR: Network connection lost.

D1_ERROR: Network connection lost.

error. Sometimes after a few seconds, sometimes within a second

ajgeiss0702•4/11/25, 12:00 AM

nvm now its fine again, seems very random when it happens

Aajgeiss0702 I switched to using the sessions api for one of my D1 DBs (that has replication ...

lambrospetrouOP•4/11/25, 12:16 AM

Network connection lost.Network connection lost. are transient errors reaching the location/datacenter of your database.

My suggestion is to add retries for some common errors. See an example in the D1 read replication demo of how to do retries: https://github.com/cloudflare/templates/blob/main/d1-starter-sessions-api-template/src/index.ts#L108
And how it's used: https://github.com/cloudflare/templates/blob/c624b31b63568adebcba1b63f600dbc81cf3a5fa/d1-starter-sessions-api-template/src/index.ts#L59

Llambrospetrou `Network connection lost.` are transient errors reaching the location/datacenter...

ajgeiss0702•4/11/25, 12:18 AM

I've gotten this error before and did retries there, but after enabling replication the error is happening way more than it did before (even though the dashboard shows that no replicas have been made yet)

ajgeiss0702•4/11/25, 12:18 AM

before, this error would only happen on some of my hottest paths on the most popular day of the week, but now its even happening today (which is a low day) on paths that don't get a ton of traffic

Aajgeiss0702 I've gotten this error before and did retries there, but after enabling replicat...

Zack•4/11/25, 12:18 AM

Have you moved your worker to use the sessions API?

Unless there is an incident the normal replication lag is a round trip from the primary to the repli

Similar Threads

Similar Threads

Similar Threads