These were all the errors we experienced from D1, which was most of our cache missed requests. I thi

These were all the errors we experienced from D1, which was most of our cache missed requests. I think a few requests successfully ran, but it was a very low percentage and the database fell over immediately afterwards again

ZackOP•4/11/25, 10:08 AM

More on the above, tried to slowly roll in the load on the replicated database, which all seemed fine until hitting 75% traffic, which has now caused p80 to rocket and errors to start appearing. Waiting to see if it dies down.

ZackOP•4/11/25, 10:09 AM

Can someone at cloudflare confirm a suspicion I have? Non-replicated D1 feels like either, the instances (DOs) live for longer before being shutdown or are just faster at starting up? Some of the problem seems to be that a region gets hit where the instance isn't available currently and that causes slowdowns while that replica comes online

ZZack These were all the errors we experienced from D1, which was most of our cache mi...

lambrospetrou•4/11/25, 10:28 AM

The team will be looking your database's errors today.

ZZack Can someone at cloudflare confirm a suspicion I have? Non-replicated D1 feels li...

lambrospetrou•4/11/25, 10:29 AM

Non-replicated D1 feels like either, the instances (DOs) live for longer before being shutdown or are just faster at starting up?

No. Replicas start up in the same way as any other database, and non-replicated databases do not live longer.

Llambrospetrou > Non-replicated D1 feels like either, the instances (DOs) live for longer befor...

ZackOP•4/11/25, 10:29 AM

Interesting, this behaviour is pretty confusing then

Llambrospetrou The team will be looking your database's errors today.

ZackOP•4/11/25, 10:29 AM

Thanks, let me know if there's anything you need from me

ZZack Interesting, this behaviour is pretty confusing then

lambrospetrou•4/11/25, 10:30 AM

Some of the problem seems to be that a region gets hit where the instance isn't available currently and that causes slowdowns while that replica comes online

This is true. If you just hit your replication-enabled database from a region for the first time, the replica there will have to start up.

ZackOP•4/11/25, 10:30 AM

Is it more useful if there is load on the database when they go to look at it, or will they be able to just look at logs and metrics from now and when I tried also yesterday

Llambrospetrou > Some of the problem seems to be that a region gets hit where the instance isn'...

ZackOP•4/11/25, 10:30 AM

Right, the confusing part to me is that we saw none of this behaviour when running 6 seperate D1 instances located within each region

ZZack Right, the confusing part to me is that we saw none of this behaviour when runni...

lambrospetrou•4/11/25, 10:31 AM

I doubt that, since your normal database would have to start up as well when first queried.

ZackOP•4/11/25, 10:31 AM

Also, this database has zero writes occurring, it's solely just reads

Llambrospetrou I doubt that, since your normal database would have to start up as well when fir...

ZackOP•4/11/25, 10:31 AM

I have metrics to prove it

ZackOP•4/11/25, 10:31 AM

It may not be the startups causing the problem, but we didn't have these problems when running 6 seperate instances

ZackOP•4/11/25, 10:32 AM

Lets see what the team find when they take a look into it

ZZack It may not be the startups causing the problem, but we didn't have _these_ probl...

lambrospetrou•4/11/25, 10:33 AM

When did you first enable replication for your database?

ZackOP•4/11/25, 10:34 AM

We have 7 databases, all with the same data. 6 to have one for each region, and the 7th in preperation for read replicas. I turned read replicas on yesterday after the release, and toggled it a couple times to see if that fixed the behaviour.

ZackOP•4/11/25, 10:34 AM

This instance primary is based in WEUR

ZackOP•4/11/25, 10:35 AM

This also gives us the ability to switch between the two setups easily with just a worker rollback

ZackOP•4/11/25, 10:38 AM

But yea, just reverted back to our old setup now, and immediately we return to no errors and latencies return back to normal ranges. Not a single request >2 seconds

ZackOP•4/11/25, 10:40 AM

ZZack But yea, just reverted back to our old setup now, and immediately we return to n...

lambrospetrou•4/11/25, 10:49 AM

The latency spikes you see is probably because those fail. Anyway, we will check the errors and see what's going on with your database.

gruntlord6•4/11/25, 12:29 PM

With the read replicas being a thing, is it possible for things like the export to not lock up the db?

Or is that not possible due to how the infrastructure is setup?

gruntlord6•4/11/25, 12:29 PM

I know we can work around it by copying to a staging database first but I figured I'd ask

gruntlord6•4/11/25, 1:20 PM

Can't you just revert?

Brayden•4/11/25, 1:23 PM

@Roopendra I've mentioned this to the team just an FYI – sorry you're experiencing this!

Brayden•4/11/25, 1:24 PM

Oh good! Is your database back up now?

Ggruntlord6 With the read replicas being a thing, is it possible for things like the export ...

lambrospetrou•4/11/25, 1:24 PM

That's the plan, but not yet. Locking is still happening at the moment.

lambrospetrou•4/11/25, 1:28 PM

FYI, disabling replicas is not instant. If replicas are created and process some of your queries, it can take a few hours for them to stop processing queries and completely disappear.

Also, if the UI is slow sometimes, you can use the PATCH API directly to manage read replication: https://developers.cloudflare.com/api/resources/d1/subresources/database/methods/edit/

The read replication mode for the database. Use 'auto' to create replicas and allow D1 automatically place them around the world, or 'disabled' to not use any database replicas (it can take a few hours for all replicas to be deleted).

Also, as I mentioned yesterday our recommendation is to first try it in your staging databases or test environments and once verified that it works for your case enable it in production.

Cloudflare API | D1 › Database › Update D1 Database Partially

Interact with Cloudflare's products and services via the Cloudflare API

gruntlord6•4/11/25, 1:46 PM

In general I would not use a beta in prodution unless I have verified its working with test projects

gruntlord6•4/11/25, 1:46 PM

I know its not helpful after the fact, but figured I would mention that

RRoopendra Yeah, my fault.

Brayden•4/11/25, 1:50 PM

I'm adding a beta label to the UI to make it more clear. We should have mentioned it's still in beta in our UI as well.

gruntlord6•4/11/25, 1:59 PM

yea, it was clear in the announcement from what I recall but UI would be nice for those who don't read that diretcly or missed that part

RRoopendra Looks like it came back up again. Thank you for your help!

lambrospetrou•4/11/25, 2:15 PM

@Roopendra FYI, we disabled replication for your database altogether for now, so even if you enable it replicas won't be created. We will remove the switch on Monday, so that you can manage it again yourself.

Llambrospetrou @Roopendra FYI, we disabled replication for your database altogether for now, s...

Milan•4/11/25, 2:27 PM

In the meantime though it would be a good idea to turn replication off for that DB if you haven't already @Roopendra

Nikolas•4/11/25, 5:06 PM

Hello, we have the exact same issue. Queries fail, dashboard fails, can't turn off replication even with the API, it returns this

{"errors":[{"code":7500,"message":"internal error; reference = e_PVSMDp_a7bf1e15234b4277b9f32a19516fdc30"}],"success":false,"messages":[],"result":null}

{"errors":[{"code":7500,"message":"internal error; reference = e_PVSMDp_a7bf1e15234b4277b9f32a19516fdc30"}],"success":false,"messages":[],"result":null}

the database id is 943219ad-705a-4a47-a87d-9b56e1b8e7ea
Thank you

Nikolas•4/11/25, 5:09 PM

Also I was wondering why nothing is mentioned in cloudflarestatus.com

Nikolas•4/11/25, 5:09 PM

is it not affecting everyone?

These were all the errors we experienced from D1, which was most of our cache missed requests. I thi

Similar Threads

Similar Threads

Similar Threads