Cloudflare Developers•15mo ago

a while back I had a db fail similarly, it wouldn't show it at all

rubenf•10/29/24, 9:01 PM

is it ok to share the id publicly ?

Chaika•10/29/24, 9:01 PM

yes

rubenf•10/29/24, 9:01 PM

cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanfOP•10/29/24, 9:01 PM

but yeah, db id and account id are fine to share

Iitsmatteomanf a while back I had a db fail similarly, it wouldn't show it at all

Chaika•10/29/24, 9:01 PM

must depend on type of failure, when my broke it (way back during the beta) just refused to load details

rubenf•10/29/24, 9:02 PM

We have moved everything from RDS to D1, and we're now a little bit concerned to say the least

CChaika must depend on type of failure, when my broke it (way back during the beta) just...

itsmatteomanfOP•10/29/24, 9:02 PM

mine had a storage issue on the node, fully collapsed and didn't show anywhere, it remained in the loading state

Rrubenf We have moved everything from RDS to D1, and we're now a little bit concerned to...

itsmatteomanfOP•10/29/24, 9:02 PM

let me escalate this for you

Rrubenf cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanfOP•10/29/24, 9:02 PM

Mind giving me the account id, too?

itsmatteomanfOP•10/29/24, 9:02 PM

That's the DB ID, right?

rubenf•10/29/24, 9:03 PM

correct, just gotta find out where to get our account id

Hugo•10/29/24, 9:03 PM

our account id is f3802b49118373bb0dd68e319c9be207

Iitsmatteomanf Yeah, my best bet is that it's an issue of the specific D1 instance... can we pi...

Walshy•10/29/24, 9:03 PM

dealing with a diff inc atm - p3 it pls

WWalshy dealing with a diff inc atm - p3 it pls

itsmatteomanfOP•10/29/24, 9:05 PM

Done!

HHugo our account id is f3802b49118373bb0dd68e319c9be207

itsmatteomanfOP•10/29/24, 9:05 PM

Thank you, escalated.

rubenf•10/29/24, 9:05 PM

itsmatteomanfOP•10/29/24, 9:07 PM

Might require D1 team manual intervention, but (won't guarantee it, of course, have no idea about internally and what the issue is) it's always been a full data recovery, if it needs to be rebuilt or something

rubenf•10/29/24, 9:08 PM

the other question that i have is regardless of issue, shouldn't time travel/recovery be possible so that we can do urgent operations if needed since when we read the blog posts there was redundancy

rubenf•10/29/24, 9:08 PM

we're fortunately not super super concerned with data loss as much as just the service being down and not possible to put back online by ourself

Rrubenf the other question that i have is regardless of issue, shouldn't time travel/rec...

itsmatteomanfOP•10/29/24, 9:11 PM

Time Travel requires the underlying instance to work, afaik... Time Travel is not a backup per se

itsmatteomanfOP•10/29/24, 9:11 PM

It's a point in time recovery

rubenf•10/29/24, 9:11 PM

My understanding is that it was diff of the WAL + snapshot from time to time no ?

itsmatteomanfOP•10/29/24, 9:11 PM

It's if you accidentally drop all tables :OMEGALUL:

Rrubenf My understanding is that it was diff of the WAL + snapshot from time to time no ...

itsmatteomanfOP•10/29/24, 9:12 PM

Yeah, but it's stored with the actual db, the backups are separate and are not accessible to the user directly

rubenf•10/29/24, 9:12 PM

ok so if we want to have production workload on this, we have to backup ourselves?

itsmatteomanfOP•10/29/24, 9:13 PM

afaik yes

itsmatteomanfOP•10/29/24, 9:13 PM

or have two synced D1s, they are always different objects, especially if they are different regions

Chaika•10/29/24, 9:13 PM

I mean no matter what platform you use, you should never trust their backups, do your own, different cloud

itsmatteomanfOP•10/29/24, 9:14 PM

And also, never trust any platform to be up 100% of the time

Chaika•10/29/24, 9:14 PM

well..

rubenf•10/29/24, 9:15 PM

Idk if I agree, the whole point of having multi AZ deployments is that you can trust services like aws to never have all regions down at the same time, and thus trust snapshot of dbs to be stored there

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

Chaika•10/29/24, 9:15 PM

ok and someone compromises your account, then what?

1984 Ford Laser•10/29/24, 9:15 PM

Bad dev practice to not have offsite/offplatform backups lmao

Chaika•10/29/24, 9:16 PM

even besides that the snapshots would still be in one place/with one provider

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

itsmatteomanfOP•10/29/24, 9:16 PM

But still, you have multiple AZ (and regions, because if we are going with 100% uptime a region can and has gone offline multiple times). D1 is a single instance.

rubenf•10/29/24, 9:17 PM

yes but I thought the whole point was that it was fine since it's actually backed up with redundancy

itsmatteomanfOP•10/29/24, 9:17 PM

d1?

rubenf•10/29/24, 9:17 PM

yes

itsmatteomanfOP•10/29/24, 9:17 PM

d1 is not redundant, it's a single object

Chaika•10/29/24, 9:18 PM

D1 (well, DOs which power them) can roam throughout multiple data centers, sort of like an AZ, but this is a startup issue with it. Kind of like if your db on an external platform had an issue with its WAL log or something, no amount of replication or redundancy would fix

Chaika•10/29/24, 9:19 PM

Like you were saying earlier with its snapshots from time to time/wal log, your storage is replicated and redundant, and while they only run on one machine, if a dc fails it can startup in a different nearby one, but this is application level. Like trying to throw redundancy at a program error

rubenf•10/29/24, 9:20 PM

https://blog.cloudflare.com/sqlite-in-durable-objects/

To solve this problem, we take advantage of our global network. Every time SQLite commits a transaction, SRS will immediately forward the change log to five "follower" machines across our network. Once at least three of these followers respond that they have received the change, SRS informs the application that the write is confirmed. (As discussed earlier, the write confirmation opens the Durable Object's "output gate", unblocking network communications to the rest of the world.)

The Cloudflare Blog

Zero-latency SQLite storage in every Durable Object

Traditional cloud storage is inherently slow because it is accessed over a network and must synchronize many clients. But what if we could instead put your application code deep into the storage layer, such that your code runs where the data is stored? Durable Objects with SQLite do just that.

Chaika•10/29/24, 9:20 PM

yup just like I was saying

itsmatteomanfOP•10/29/24, 9:21 PM

This is most likely a a corrupted object, which prevents it from starting at all... so yeah, the data is backed up, but it's not able to start

Chaika•10/29/24, 9:21 PM

D1_ERROR: Internal error while starting up D1 storage caused object to be reset

this is a startup error though. It has the storage, it's replicated or whatever, but it can't run it. It doesn't matter how many machines it can fail over to, or how deeply replicated the data is, if it can't do the equivalent of loading the database

rubenf•10/29/24, 9:22 PM

ok so it raises the next question, how can a d1 database be corrupted

Chaika•10/29/24, 9:22 PM

corrupted might be a bit far, back during the beta I had that exact error msg and they said it had gone too long without snapshotting and was trying to replay too much data

Chaika•10/29/24, 9:23 PM

could be a number of issues, but in general it just can't load and get running

Chaika•10/29/24, 9:23 PM

it shouldn't happen regardless of anything else, off-site/diff provider backups are a good idea unrelated to this though, hopefully they respond to esc soon

rubenf•10/29/24, 9:34 PM

Thanks, we build distributed systems ourselves (we're behind: https://github.com/windmill-labs/windmill) and if we told our paid customers that they should backup themselves their cloud hosted instance, it wouldn't go very well as that's what our customers pay us to do, think very hard how to make distributed systems invulnerable so that they don't have to. We only store billing info on D1, the rest runs on AWS, I do think the point of those services is that those situation never happen. I'm very excited by D1, it's great tech, it just may be a bit early on some aspects.

a while back I had a db fail similarly, it wouldn't show it at all

Similar Threads

Similar Threads

Similar Threads