Cloudflare Developers•9mo ago

i'm still getting errors occasionally with 5 retries

Aajgeiss0702 i'm still getting errors occasionally with 5 retries

Milan•4/12/25, 3:41 PM

For replication? We are working on a couple of bug fixes we're hoping to get out midway through this coming week

Deltachaos•4/12/25, 11:19 PM

regarding the limits: if i want to run a query like this:
update foo set bar = 1 where id not in (?)update foo set bar = 1 where id not in (?)
how do i bind a list as paramter? or do i need to do this?
update foo set bar = 1 where id not in (?, ?, ?, ?)update foo set bar = 1 where id not in (?, ?, ?, ?)

DDeltachaos regarding the limits: if i want to run a query like this: `update foo set bar = ...

lambrospetrou•4/13/25, 6:40 AM

The latter.

MMilan For replication? We are working on a couple of bug fixes we're hoping to get out...

ajgeiss0702OP•4/13/25, 8:18 PM

thank you. Will there be any kind of notification when these fixes are rolled out? Or should I just keep an eye on this channel

kapitein syantje•4/13/25, 8:58 PM

Hi, my production database was accidentally deleted whilst executing migrations. Not sure how it happened. I have the ID of the production database as well as the binding, is there a way to revert this?

Kkapitein syantje Hi, my production database was accidentally deleted whilst executing migrations....

Milan•4/13/25, 9:25 PM

I don't work on D1 but I would assume https://developers.cloudflare.com/d1/reference/time-travel/ is a good place to start, otherwise open a support ticket?

Cloudflare Docs

Time Travel and backups · Cloudflare D1 docs

Time Travel is D1's approach to backups and point-in-time-recovery, and allows you to restore a database to any minute within the last 30 days.

Aajgeiss0702 thank you. Will there be any kind of notification when these fixes are rolled ou...

Milan•4/13/25, 9:26 PM

I would probably say keep an eye on this channel (we will make noise), but also we'll probably remove some banners we've added around the UI here and there

MMilan I don't work on D1 but I would assume https://developers.cloudflare.com/d1/refer...

kapitein syantje•4/13/25, 9:26 PM

Hi Milan, thank you for your response. My first bet was to use time travel indeed, however, I cannot reference the database as it was deleted. I cannot time travel if the DB is deleted in Cloudflare apparantly.

Hopefully nothing is permanent, not the most fun to experience at midnight

MMilan I would probably say keep an eye on this channel (we will make noise), but also ...

ajgeiss0702OP•4/13/25, 9:27 PM

ok, thank you. I'll try to check this channel every few days

I've basically already been doing that anyway

Kkapitein syantje Hi Milan, thank you for your response. My first bet was to use time travel indee...

Milan•4/13/25, 9:29 PM

You mean you went into the UI/API and deleted the DB outright?

MMilan You mean you went into the UI/API and deleted the DB outright?

kapitein syantje•4/13/25, 9:30 PM

I suppose so? I was simply doing migrations using drizzle, and it probably did an API call deleting the db. I did not directly delete the DB through UI/API

kapitein syantje•4/13/25, 9:34 PM

Can confirm an API request went through, just checked the audit log... very curious

Milan•4/13/25, 9:35 PM

I've started a thread internally, is this DB critical for your service?

kapitein syantje•4/13/25, 9:36 PM

Very much so I'm afraid. The service runs mainly on the D1 db.

Kkapitein syantje Very much so I'm afraid. The service runs mainly on the D1 db.

Milan•4/13/25, 9:40 PM

Mind DMing me your DB ID + account ID?

Milan•4/13/25, 9:40 PM

Might help expedite things.

Chaika•4/14/25, 1:17 PM

I see the warning in the dash now, should help, ty d1 team :MeowHeartCloudflare:

cosbgn•4/14/25, 1:36 PM

Activating D1 replica without any code/change so without using the session API has no benefit right?

Chaika•4/14/25, 1:39 PM

right, all your queries would still go to primary

Chaika•4/14/25, 1:40 PM

you can use the sessions api w/o replication enabled though, to get your code ready

Ccosbgn Activating D1 replica without any code/change so without using the session API h...

lambrospetrou•4/14/25, 1:55 PM

Functionally no, but in practice there could be a regression depending on your traffic and workload, similar to the issue some users already faced. Our fix for that specific issue is pending rollout.
Your queries will probably flow through replicas and alwas forwarded to the primary. So, there is a difference in practice.

Chaika•4/14/25, 4:17 PM

Just sharing some feedback about read replicas here publicly:
Read Replicas are super cool and long awaited, when they work well they do a good job of making queries low latency world wide, and sessions api isn't too hard to use in my opinion.

But I think they're a bit hard to recommend for most use cases right now, for a few reasons:

In the shorter term, there's a good amount of errors with them. I've gotten nearly 10k errors in the last 2 days or so, like 10% error rate or so on queries, from simple select queries. Mostly network connection lost and internal errors. D1 team has said they're aware, but it's still painful to use with, need lots of retries. Don't think database specific, seen this across 3 super simple DBs.

I'm sure that'll be fixed with time, but the bigger issue I see is with cold starts. D1 has always had cold starts with 30-40 seconds of inactivity, pretty quick, that results in incurring a cold start on the next query. However, they seem worse with read replicas.
It seems to be location dependent, for a db with primary in enam (~16 KBs or so size)
enam replica cold start - 1s
eeur replica cold start - 2-3s
apac replica cold start - 4-5s
normal non-replication cold start for same db: ~300-400ms

Perhaps because it's re-syncing first before answering traffic? I would think it could revive the replica quickly and sync async, but perhaps it doesn't work that way? It'd be really nice if either the keep alive without any request was longer (like 10m-15m) and/or the cold start was reduced.

Unless you have high RPS world-wide, I think currently you'd be better off with read replicas off and keeping your primary warm (or incurring its much smaller cold starts). Even if you have active traffic in EEUR with an ENAM Primary, infrequent requests from APAC will still be painful for users. Maybe more precise replica configs could help that too.

It's beta though, and I'm sure with some more love it'll be super useful to have on for workers apps soon.

CChaika Just sharing some feedback about read replicas here publicly: Read Replicas are ...

lambrospetrou•4/14/25, 4:46 PM

Thanks a lot for the honest and detailed feedback. The good thing is that we are aware of all the things you mentioned. The number of issues introduced with replicas will get back to normal levels hopefully by end of the week. (Not saying D1 in general doesn't have more errors than what people want - we are working on that too)

Better and smarter replica placement is on the roadmap as well.

Llambrospetrou Thanks a lot for the honest and detailed feedback. The good thing is that we are...

lambrospetrou•4/14/25, 4:51 PM

Maybe more precise replica configs could help that too.

For the sake of brainstorming, if you have low traffic, say 1 request every 15 minutes from some regions, how would a replica config help? How do you imagine it behaving from your side?

Llambrospetrou > Maybe more precise replica configs could help that too. For the sake of brains...

Chaika•4/14/25, 4:52 PM

I meant that more in the case of: Having lots of traffic in ENAM/EEUR (so replicas in both, kept warm/active), and being able to disable Oceanic and Asia Pacific replica regions so infrequent requests from those regions don't incur the currently quite large 4-5 seconds of cold start latency over and over

CChaika I meant that more in the case of: Having lots of traffic in ENAM/EEUR (so replic...

lambrospetrou•4/14/25, 4:54 PM

Ah yes, this makes sense.

Chaika•4/14/25, 4:54 PM

Although I suppose the other way to solve that would be to lower the cold start time in those far away regions from your primary, not sure what would be involved for that though

Zack•4/14/25, 5:14 PM

I mean, would another option to be able to pay an extra fee for "always-on" replicas?

Zack•4/14/25, 5:15 PM

Doing rough maths, replicas are essentially a durable object, keeping a durable object online for an entire month would normally cost just over $4 with duration costs

Zack•4/14/25, 5:16 PM

Obviously reducing replica cold start time is the preference so that isn't necessary, but maybe for people who don't want to deal with that at all, having an "always-on" option would be something that could be considered

ZZack I mean, would another option to be able to pay an extra fee for "always-on" repl...

lambrospetrou•4/14/25, 5:45 PM

You can do this easily already by having a DO doing

select 1;

select 1;

from each region with an alarm every 1 minute if you really want, but at this point there isn't much gained from using a replica. Is that single request every few minutes that important that you cannot go to the primary for an extra 100-200ms? We need to provide a way to skip replicas and go straight to the primary though to avoid the replica booting up.

Llambrospetrou You can do this easily already by having a DO doing `select 1;` from each region...

Zack•4/14/25, 5:45 PM

Depends how quickly those instances shutdown, but if you're on the other side of the world to the primary, probably, yes

Zack•4/14/25, 5:46 PM

I did consider doing the SELECT 1 to keep it warm, but obviously something official would be nicer. Surprised you guys are suggesting something like that publicly

Zack•4/14/25, 5:47 PM

I'll be giving it another go once these fixes are in for the issues last week, maybe after that it won't be necessary anyway

Zack•4/14/25, 5:54 PM

It also depends on your traffic patterns, for us for example, we have lower traffic in APAC and OC, especially overnight, but when those requests do come through it can be anywhere between 10 or 100's of requests

Zack•4/14/25, 5:54 PM

For those regions it probably takes >1 second to get to the primary

Zack•4/14/25, 5:55 PM

FWIW though we don't currently see problematic behaviour wrt this on our existing databases, which are one per region

Zack•4/14/25, 5:57 PM

I don't know if you can share this, but do the replicas have to go back to the primary storage in the primary region to retrieve the database when cold starting? Or are some checkpoints stored local to that region, so that it only has to sync what has changed since last time via the WAL?

ZZack I did consider doing the SELECT 1 to keep it warm, but obviously something offic...

lambrospetrou•4/14/25, 6:36 PM

I am NOT suggesting it. I said you can do it already if that's what you want.

Llambrospetrou I am NOT suggesting it. I said you can do it already if that's what you want.

Zack•4/14/25, 6:36 PM

I more meant even saying it publicly

ZZack I more meant even saying it publicly 🤣

lambrospetrou•4/14/25, 6:37 PM

I mean, you pay for it.

Zack•4/14/25, 6:37 PM

Sure, reads are essentially free though

Zack•4/14/25, 6:38 PM

I guess if you do it from DOs you're paying for keeping those alive

Matthew•4/14/25, 9:44 PM

Hey, has anybody done any measurement/comparison between D1 and Hyperdrive performance wise? I'd like to migrate from D1 to Hyperdrive and would love to know more beforehand.
Thanks Xx

MMatthew Hey, has anybody done any measurement/comparison between D1 and Hyperdrive perfo...

lambrospetrou•4/14/25, 9:46 PM

Without specifying your workload (volume + kind of queries), location of source and target, and latency needs, no benchmark would really make sense for you.

Llambrospetrou Without specifying your workload (volume + kind of queries), location of source ...

Matthew•4/14/25, 9:47 PM

I see, you're right. But it would generally help if we knew which one would be better for what type of use cases..

Llambrospetrou Without specifying your workload (volume + kind of queries), location of source ...

lambrospetrou•4/14/25, 9:48 PM

For example, if all your users are in EEUR/WEUR, and you use Hyperdrive with a Postgres database on a 4GB host in Frankfurt, that's probably going to perform amazingly well, and probably better than a single D1 database.
If you use tens of D1 databases spread around close to each of your user, and you have traffic from everywhere, then the comparion is more nuanced.

jed•4/15/25, 3:00 AM

is there a performance penalty for using D1 via the REST API from a worker, as opposed to using the Workers Binding API? or is the binding just a wrapper around the REST API?

Jjed is there a performance penalty for using D1 via the REST API from a worker, as o...

Cyb3r-Jak3•4/15/25, 3:04 AM

Yes. All requests have to go back to Cloudflare's core, so request latency is far from optimal from some locations. You would also be subject to the API Rate limit of 1200 requests per 5 minutes.

CCyb3r-Jak3 Yes. All requests have to go back to Cloudflare's core, so request latency is fa...

jed•4/15/25, 3:05 AM

awesome, thank you!

i'm still getting errors occasionally with 5 retries

Similar Threads

Similar Threads

Similar Threads