Cloudflare Developers•15mo ago

We have moved everything from RDS to D1, and we're now a little bit concerned to say the least

CChaika must depend on type of failure, when my broke it (way back during the beta) just...

itsmatteomanf•10/29/24, 9:02 PM

mine had a storage issue on the node, fully collapsed and didn't show anywhere, it remained in the loading state

Rrubenf We have moved everything from RDS to D1, and we're now a little bit concerned to...

itsmatteomanf•10/29/24, 9:02 PM

let me escalate this for you

Rrubenf cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanf•10/29/24, 9:02 PM

Mind giving me the account id, too?

itsmatteomanf•10/29/24, 9:02 PM

That's the DB ID, right?

rubenfOP•10/29/24, 9:03 PM

correct, just gotta find out where to get our account id

Hugo•10/29/24, 9:03 PM

our account id is f3802b49118373bb0dd68e319c9be207

Iitsmatteomanf Yeah, my best bet is that it's an issue of the specific D1 instance... can we pi...

Walshy•10/29/24, 9:03 PM

dealing with a diff inc atm - p3 it pls

WWalshy dealing with a diff inc atm - p3 it pls

itsmatteomanf•10/29/24, 9:05 PM

Done!

HHugo our account id is f3802b49118373bb0dd68e319c9be207

itsmatteomanf•10/29/24, 9:05 PM

Thank you, escalated.

rubenfOP•10/29/24, 9:05 PM

itsmatteomanf•10/29/24, 9:07 PM

Might require D1 team manual intervention, but (won't guarantee it, of course, have no idea about internally and what the issue is) it's always been a full data recovery, if it needs to be rebuilt or something

rubenfOP•10/29/24, 9:08 PM

the other question that i have is regardless of issue, shouldn't time travel/recovery be possible so that we can do urgent operations if needed since when we read the blog posts there was redundancy

rubenfOP•10/29/24, 9:08 PM

we're fortunately not super super concerned with data loss as much as just the service being down and not possible to put back online by ourself

Rrubenf the other question that i have is regardless of issue, shouldn't time travel/rec...

itsmatteomanf•10/29/24, 9:11 PM

Time Travel requires the underlying instance to work, afaik... Time Travel is not a backup per se

itsmatteomanf•10/29/24, 9:11 PM

It's a point in time recovery

rubenfOP•10/29/24, 9:11 PM

My understanding is that it was diff of the WAL + snapshot from time to time no ?

itsmatteomanf•10/29/24, 9:11 PM

It's if you accidentally drop all tables :OMEGALUL:

Rrubenf My understanding is that it was diff of the WAL + snapshot from time to time no ...

itsmatteomanf•10/29/24, 9:12 PM

Yeah, but it's stored with the actual db, the backups are separate and are not accessible to the user directly

rubenfOP•10/29/24, 9:12 PM

ok so if we want to have production workload on this, we have to backup ourselves?

itsmatteomanf•10/29/24, 9:13 PM

afaik yes

itsmatteomanf•10/29/24, 9:13 PM

or have two synced D1s, they are always different objects, especially if they are different regions

Chaika•10/29/24, 9:13 PM

I mean no matter what platform you use, you should never trust their backups, do your own, different cloud

itsmatteomanf•10/29/24, 9:14 PM

And also, never trust any platform to be up 100% of the time

Chaika•10/29/24, 9:14 PM

well..

rubenfOP•10/29/24, 9:15 PM

Idk if I agree, the whole point of having multi AZ deployments is that you can trust services like aws to never have all regions down at the same time, and thus trust snapshot of dbs to be stored there

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

Chaika•10/29/24, 9:15 PM

ok and someone compromises your account, then what?

1984 Ford Laser•10/29/24, 9:15 PM

Bad dev practice to not have offsite/offplatform backups lmao

Chaika•10/29/24, 9:16 PM

even besides that the snapshots would still be in one place/with one provider

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

itsmatteomanf•10/29/24, 9:16 PM

But still, you have multiple AZ (and regions, because if we are going with 100% uptime a region can and has gone offline multiple times). D1 is a single instance.

rubenfOP•10/29/24, 9:17 PM

yes but I thought the whole point was that it was fine since it's actually backed up with redundancy

itsmatteomanf•10/29/24, 9:17 PM

d1?

rubenfOP•10/29/24, 9:17 PM

yes

itsmatteomanf•10/29/24, 9:17 PM

d1 is not redundant, it's a single object

Chaika•10/29/24, 9:18 PM

D1 (well, DOs which power them) can roam throughout multiple data centers, sort of like an AZ, but this is a startup issue with it. Kind of like if your db on an external platform had an issue with its WAL log or something, no amount of replication or redundancy would fix

Chaika•10/29/24, 9:19 PM

Like you were saying earlier with its snapshots from time to time/wal log, your storage is replicated and redundant, and while they only run on one machine, if a dc fails it can startup in a different nearby one, but this is application level. Like trying to throw redundancy at a program error

rubenfOP•10/29/24, 9:20 PM

https://blog.cloudflare.com/sqlite-in-durable-objects/

To solve this problem, we take advantage of our global network. Every time SQLite commits a transaction, SRS will immediately forward the change log to five "follower" machines across our network. Once at least three of these followers respond that they have received the change, SRS informs the application that the write is confirmed. (As discussed earlier, the write confirmation opens the Durable Object's "output gate", unblocking network communications to the rest of the world.)

The Cloudflare Blog

Zero-latency SQLite storage in every Durable Object

Traditional cloud storage is inherently slow because it is accessed over a network and must synchronize many clients. But what if we could instead put your application code deep into the storage layer, such that your code runs where the data is stored? Durable Objects with SQLite do just that.

Chaika•10/29/24, 9:20 PM

yup just like I was saying

itsmatteomanf•10/29/24, 9:21 PM

This is most likely a a corrupted object, which prevents it from starting at all... so yeah, the data is backed up, but it's not able to start

Chaika•10/29/24, 9:21 PM

D1_ERROR: Internal error while starting up D1 storage caused object to be reset

this is a startup error though. It has the storage, it's replicated or whatever, but it can't run it. It doesn't matter how many machines it can fail over to, or how deeply replicated the data is, if it can't do the equivalent of loading the database

rubenfOP•10/29/24, 9:22 PM

ok so it raises the next question, how can a d1 database be corrupted

Chaika•10/29/24, 9:22 PM

corrupted might be a bit far, back during the beta I had that exact error msg and they said it had gone too long without snapshotting and was trying to replay too much data

Chaika•10/29/24, 9:23 PM

could be a number of issues, but in general it just can't load and get running

Chaika•10/29/24, 9:23 PM

it shouldn't happen regardless of anything else, off-site/diff provider backups are a good idea unrelated to this though, hopefully they respond to esc soon

rubenfOP•10/29/24, 9:34 PM

Thanks, we build distributed systems ourselves (we're behind: https://github.com/windmill-labs/windmill) and if we told our paid customers that they should backup themselves their cloud hosted instance, it wouldn't go very well as that's what our customers pay us to do, think very hard how to make distributed systems invulnerable so that they don't have to. We only store billing info on D1, the rest runs on AWS, I do think the point of those services is that those situation never happen. I'm very excited by D1, it's great tech, it just may be a bit early on some aspects.

1984 Ford Laser•10/29/24, 9:38 PM

But we don't pay CF to backup our data

rubenfOP•10/29/24, 9:39 PM

I pay CF to build a reliable service and have quick fallback when they go down, otherwise what's the point

rubenfOP•10/29/24, 9:40 PM

We're happy to pay more, I understand that production critical services is not the intended audience just yet without extra machinery

1984 Ford Laser•10/29/24, 9:40 PM

I mean the service is overall pretty reliable, and the team has been very quick to respond to your issue, at least to onboard the issue. They have also been very realistic about the platform, where no one can ever guarantee 100% uptime (I understand the SLA and stuff)

1984 Ford Laser•10/29/24, 9:41 PM

Paying more isn't the point tbh. This platform is designed to let you do what you want, which means you need to bring your own batteries, and sometimes the motherboard and storage too, but the CPU is there and its cheap and feature aplenty

rubenfOP•10/29/24, 9:42 PM

I mean some part of our billing service is down for an hour now, and no we just know that a ticket has been raised. It's nice but it's not great. It's cheap, that's for sure, I was hoping it was more than that.

We have moved everything from RDS to D1, and we're now a little bit concerned to say the least

Similar Threads

Similar Threads

Similar Threads