I want to stress this again. Do not deploy beta products to your production service. It's in beta fo

I want to stress this again. Do not deploy beta products to your production service. It's in beta for a reason. @lambrospetrou can we pin this message somewhere?

Nikolas•4/11/25, 6:51 PM

but it didnt say beta when we enabled it...

Nikolas•4/11/25, 6:52 PM

we had no idea you launched this yesterday

Nikolas•4/11/25, 6:52 PM

and it said it would just improve the db latency so why not enable it?

Original message was deleted

Isaac McFadyen•4/11/25, 6:55 PM

IIRC that's newly added, it wasn't there at launch.

Nikolas•4/11/25, 6:55 PM

yes

Nikolas•4/11/25, 6:55 PM

it wasn't there

Nikolas•4/11/25, 6:59 PM

also someone would assume that an option can be disabled if needed.. Wrong assumption I guess

MMilan I want to stress this again. Do not deploy beta products to your production serv...

Chaika•4/11/25, 7:06 PM

should perhaps be a popup on enabling it, asking for confirmation "Read Replication is in beta and can take hours to disable after enabling. Only enable for non-production/in testing" or something like that

Chaika•4/11/25, 7:06 PM

it'd be way less of an issue if disabling it was instant/always worked

MilanOP•4/11/25, 7:08 PM

Disabling does work. We're currently working on fixes for a few known bugs (some new ones as of going public), your DB hit one of them @Nikolas. To be clear, not all replicated DBs will hit these problems, but when DBs do hit these problems we intend to patch the issues as quickly as we can.

As is mentioned in the blog, it's really important to verify that these things work for your use case prior to releasing them to production. Even with products that are GA, it's critical to understand what you're releasing.

MMilan I want to stress this again. Do not deploy beta products to your production serv...

Chaika•4/11/25, 7:15 PM

we can pin messages in channels. Do you want just that message pinned, or something more exact?

Chaika•4/11/25, 7:16 PM

like "D1 Read Replication is in beta and not recommended for production use. Exercise care when enabling before releasing to production"?

CChaika we can pin messages in channels. Do you want just that message pinned, or someth...

lambrospetrou•4/11/25, 7:19 PM

D1 Read replication is in BETA. Do not enable directly in production without testing in a staging/test environment for a few days to ensure it works fine for your workload. The D1 team is aware of certain issues leading to continuous errors for some users, and are working on a fix.

AtesMAE•4/11/25, 7:19 PM

I have a problem, where can I get support

James•4/11/25, 7:29 PM

What kind of problem? What do you need support for? If it's for D1 as per this channel, you can simply ask your question or state your problem and someone will get back to you. If it's something else, check the other channels in the server.

AtesMAE•4/11/25, 7:30 PM

could not run legacy post zone sub request against new zone products: failed to update user subscription: failed to apply object products: You cannot add or modify subscriptions or services until the outstanding balance is paid. You should be able to do so in your Billing page.

AtesMAE•4/11/25, 7:30 PM

i need help

AAtesMAE could not run legacy post zone sub request against new zone products: failed to ...

Isaac McFadyen•4/11/25, 7:31 PM

?billing See below

SuperHelpflare•4/11/25, 7:31 PM

Cloudflare's billing system is undergoing some changes/migration. During this period, you may notice extra bills, incorrect bills, or other issues.

To resolve these issues, create a Support ticket by using the following link (you must be logged into your Cloudflare account): https://dash.cloudflare.com/?to=/:account/support

Note that all accounts, even Free plan accounts, can create billing tickets. Tickets are the only way to resolve billing issues; we cannot help you in this Discord.

ajgeiss0702•4/12/25, 2:49 AM

i'm still getting errors occasionally with 5 retries

Aajgeiss0702 i'm still getting errors occasionally with 5 retries

MilanOP•4/12/25, 3:41 PM

For replication? We are working on a couple of bug fixes we're hoping to get out midway through this coming week

Deltachaos•4/12/25, 11:19 PM

regarding the limits: if i want to run a query like this:
update foo set bar = 1 where id not in (?)update foo set bar = 1 where id not in (?)
how do i bind a list as paramter? or do i need to do this?
update foo set bar = 1 where id not in (?, ?, ?, ?)update foo set bar = 1 where id not in (?, ?, ?, ?)

DDeltachaos regarding the limits: if i want to run a query like this: `update foo set bar = ...

lambrospetrou•4/13/25, 6:40 AM

The latter.

MMilan For replication? We are working on a couple of bug fixes we're hoping to get out...

ajgeiss0702•4/13/25, 8:18 PM

thank you. Will there be any kind of notification when these fixes are rolled out? Or should I just keep an eye on this channel

kapitein syantje•4/13/25, 8:58 PM

Hi, my production database was accidentally deleted whilst executing migrations. Not sure how it happened. I have the ID of the production database as well as the binding, is there a way to revert this?

Kkapitein syantje Hi, my production database was accidentally deleted whilst executing migrations....

MilanOP•4/13/25, 9:25 PM

I don't work on D1 but I would assume https://developers.cloudflare.com/d1/reference/time-travel/ is a good place to start, otherwise open a support ticket?

Cloudflare Docs

Time Travel and backups · Cloudflare D1 docs

Time Travel is D1's approach to backups and point-in-time-recovery, and allows you to restore a database to any minute within the last 30 days.

Aajgeiss0702 thank you. Will there be any kind of notification when these fixes are rolled ou...

MilanOP•4/13/25, 9:26 PM

I would probably say keep an eye on this channel (we will make noise), but also we'll probably remove some banners we've added around the UI here and there

MMilan I don't work on D1 but I would assume https://developers.cloudflare.com/d1/refer...

kapitein syantje•4/13/25, 9:26 PM

Hi Milan, thank you for your response. My first bet was to use time travel indeed, however, I cannot reference the database as it was deleted. I cannot time travel if the DB is deleted in Cloudflare apparantly.

Hopefully nothing is permanent, not the most fun to experience at midnight

MMilan I would probably say keep an eye on this channel (we will make noise), but also ...

ajgeiss0702•4/13/25, 9:27 PM

ok, thank you. I'll try to check this channel every few days

I've basically already been doing that anyway

Kkapitein syantje Hi Milan, thank you for your response. My first bet was to use time travel indee...

MilanOP•4/13/25, 9:29 PM

You mean you went into the UI/API and deleted the DB outright?

MMilan You mean you went into the UI/API and deleted the DB outright?

kapitein syantje•4/13/25, 9:30 PM

I suppose so? I was simply doing migrations using drizzle, and it probably did an API call deleting the db. I did not directly delete the DB through UI/API

kapitein syantje•4/13/25, 9:34 PM

Can confirm an API request went through, just checked the audit log... very curious

MilanOP•4/13/25, 9:35 PM

I've started a thread internally, is this DB critical for your service?

kapitein syantje•4/13/25, 9:36 PM

Very much so I'm afraid. The service runs mainly on the D1 db.

Kkapitein syantje Very much so I'm afraid. The service runs mainly on the D1 db.

MilanOP•4/13/25, 9:40 PM

Mind DMing me your DB ID + account ID?

MilanOP•4/13/25, 9:40 PM

Might help expedite things.

Chaika•4/14/25, 1:17 PM

I see the warning in the dash now, should help, ty d1 team :MeowHeartCloudflare:

cosbgn•4/14/25, 1:36 PM

Activating D1 replica without any code/change so without using the session API has no benefit right?

Chaika•4/14/25, 1:39 PM

right, all your queries would still go to primary

Chaika•4/14/25, 1:40 PM

you can use the sessions api w/o replication enabled though, to get your code ready

Ccosbgn Activating D1 replica without any code/change so without using the session API h...

lambrospetrou•4/14/25, 1:55 PM

Functionally no, but in practice there could be a regression depending on your traffic and workload, similar to the issue some users already faced. Our fix for that specific issue is pending rollout.
Your queries will probably flow through replicas and alwas forwarded to the primary. So, there is a difference in practice.

Chaika•4/14/25, 4:17 PM

Just sharing some feedback about read replicas here publicly:
Read Replicas are super cool and long awaited, when they work well they do a good job of making queries low latency world wide, and sessions api isn't too hard to use in my opinion.

But I think they're a bit hard to recommend for most use cases right now, for a few reasons:

In the shorter term, there's a good amount of errors with them. I've gotten nearly 10k errors in the last 2 days or so, like 10% error rate or so on queries, from simple select queries. Mostly network connection lost and internal errors. D1 team has said they're aware, but it's still painful to use with, need lots of retries. Don't think database specific, seen this across 3 super simple DBs.

I'm sure that'll be fixed with time, but the bigger issue I see is with cold starts. D1 has always had cold starts with 30-40 seconds of inactivity, pretty quick, that results in incurring a cold start on the next query. However, they seem worse with read replicas.
It seems to be location dependent, for a db with primary in enam (~16 KBs or so size)
enam replica cold start - 1s
eeur replica cold start - 2-3s
apac replica cold start - 4-5s
normal non-replication cold start for same db: ~300-400ms

Perhaps because it's re-syncing first before answering traffic? I would think it could revive the replica quickly and sync async, but perhaps it doesn't work that way? It'd be really nice if either the keep alive without any request was longer (like 10m-15m) and/or the cold start was reduced.

Unless you have high RPS world-wide, I think currently you'd be better off with read replicas off and keeping your primary warm (or incurring its much smaller cold starts). Even if you have active traffic in EEUR with an ENAM Primary, infrequent requests from APAC will still be painful for users. Maybe more precise replica configs could help that too.

It's beta though, and I'm sure with some more love it'll be super useful to have on for workers apps soon.

CChaika Just sharing some feedback about read replicas here publicly: Read Replicas are ...

lambrospetrou•4/14/25, 4:46 PM

Thanks a lot for the honest and detailed feedback. The good thing is that we are aware of all the things you mentioned. The number of issues introduced with replicas will get back to normal levels hopefully by end of the week. (Not saying D1 in general doesn't have more errors than what people want - we are working on that too)

Better and smarter replica placement is on the roadmap as well.

Llambrospetrou Thanks a lot for the honest and detailed feedback. The good thing is that we are...

lambrospetrou•4/14/25, 4:51 PM

Maybe more precise replica configs could help that too.

For the sake of brainstorming, if you have low traffic, say 1 request every 15 minutes from some regions, how would a replica config help? How do you imagine it behaving from your side?

Llambrospetrou > Maybe more precise replica configs could help that too. For the sake of brains...

Chaika•4/14/25, 4:52 PM

I meant that more in the case of: Having lots of traffic in ENAM/EEUR (so replicas in both, kept warm/active), and being able to disable Oceanic and Asia Pacific replica regions so infrequent requests from those regions don't incur the currently quite large 4-5 seconds of cold start latency over and over

CChaika I meant that more in the case of: Having lots of traffic in ENAM/EEUR (so replic...

lambrospetrou•4/14/25, 4:54 PM

Ah yes, this makes sense.

Chaika•4/14/25, 4:54 PM

Although I suppose the other way to solve that would be to lower the cold start time in those far away regions from your primary, not sure what would be involved for that though

Zack•4/14/25, 5:14 PM

I mean, would another option to be able to pay an extra fee for "always-on" replicas?

Zack•4/14/25, 5:15 PM

Doing rough maths, replicas are essentially a durable object, keeping a durable object online for an entire month would normally cost just over $4 with duration costs

Zack•4/14/25, 5:16 PM

Obviously reducing replica cold start time is the preference so that isn't necessary, but maybe for people who don't want to deal with that at all, having an "always-on" option would be something that could be considered

I want to stress this again. Do not deploy beta products to your production service. It's in beta fo

Similar Threads

Similar Threads

Similar Threads