Idk if I agree, the whole point of having multi AZ deployments is that you can trust services like a

Idk if I agree, the whole point of having multi AZ deployments is that you can trust services like aws to never have all regions down at the same time, and thus trust snapshot of dbs to be stored there

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

Chaika•10/29/24, 9:15 PM

ok and someone compromises your account, then what?

1984 Ford Laser•10/29/24, 9:15 PM

Bad dev practice to not have offsite/offplatform backups lmao

Chaika•10/29/24, 9:16 PM

even besides that the snapshots would still be in one place/with one provider

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

itsmatteomanf•10/29/24, 9:16 PM

But still, you have multiple AZ (and regions, because if we are going with 100% uptime a region can and has gone offline multiple times). D1 is a single instance.

rubenfOP•10/29/24, 9:17 PM

yes but I thought the whole point was that it was fine since it's actually backed up with redundancy

itsmatteomanf•10/29/24, 9:17 PM

d1?

rubenfOP•10/29/24, 9:17 PM

yes

itsmatteomanf•10/29/24, 9:17 PM

d1 is not redundant, it's a single object

Chaika•10/29/24, 9:18 PM

D1 (well, DOs which power them) can roam throughout multiple data centers, sort of like an AZ, but this is a startup issue with it. Kind of like if your db on an external platform had an issue with its WAL log or something, no amount of replication or redundancy would fix

Chaika•10/29/24, 9:19 PM

Like you were saying earlier with its snapshots from time to time/wal log, your storage is replicated and redundant, and while they only run on one machine, if a dc fails it can startup in a different nearby one, but this is application level. Like trying to throw redundancy at a program error

rubenfOP•10/29/24, 9:20 PM

https://blog.cloudflare.com/sqlite-in-durable-objects/

To solve this problem, we take advantage of our global network. Every time SQLite commits a transaction, SRS will immediately forward the change log to five "follower" machines across our network. Once at least three of these followers respond that they have received the change, SRS informs the application that the write is confirmed. (As discussed earlier, the write confirmation opens the Durable Object's "output gate", unblocking network communications to the rest of the world.)

The Cloudflare Blog

Zero-latency SQLite storage in every Durable Object

Traditional cloud storage is inherently slow because it is accessed over a network and must synchronize many clients. But what if we could instead put your application code deep into the storage layer, such that your code runs where the data is stored? Durable Objects with SQLite do just that.

Chaika•10/29/24, 9:20 PM

yup just like I was saying

itsmatteomanf•10/29/24, 9:21 PM

This is most likely a a corrupted object, which prevents it from starting at all... so yeah, the data is backed up, but it's not able to start

Chaika•10/29/24, 9:21 PM

D1_ERROR: Internal error while starting up D1 storage caused object to be reset

this is a startup error though. It has the storage, it's replicated or whatever, but it can't run it. It doesn't matter how many machines it can fail over to, or how deeply replicated the data is, if it can't do the equivalent of loading the database

rubenfOP•10/29/24, 9:22 PM

ok so it raises the next question, how can a d1 database be corrupted

Chaika•10/29/24, 9:22 PM

corrupted might be a bit far, back during the beta I had that exact error msg and they said it had gone too long without snapshotting and was trying to replay too much data

Chaika•10/29/24, 9:23 PM

could be a number of issues, but in general it just can't load and get running

Chaika•10/29/24, 9:23 PM

it shouldn't happen regardless of anything else, off-site/diff provider backups are a good idea unrelated to this though, hopefully they respond to esc soon

rubenfOP•10/29/24, 9:34 PM

Thanks, we build distributed systems ourselves (we're behind: https://github.com/windmill-labs/windmill) and if we told our paid customers that they should backup themselves their cloud hosted instance, it wouldn't go very well as that's what our customers pay us to do, think very hard how to make distributed systems invulnerable so that they don't have to. We only store billing info on D1, the rest runs on AWS, I do think the point of those services is that those situation never happen. I'm very excited by D1, it's great tech, it just may be a bit early on some aspects.

1984 Ford Laser•10/29/24, 9:38 PM

But we don't pay CF to backup our data

rubenfOP•10/29/24, 9:39 PM

I pay CF to build a reliable service and have quick fallback when they go down, otherwise what's the point

rubenfOP•10/29/24, 9:40 PM

We're happy to pay more, I understand that production critical services is not the intended audience just yet without extra machinery

1984 Ford Laser•10/29/24, 9:40 PM

I mean the service is overall pretty reliable, and the team has been very quick to respond to your issue, at least to onboard the issue. They have also been very realistic about the platform, where no one can ever guarantee 100% uptime (I understand the SLA and stuff)

1984 Ford Laser•10/29/24, 9:41 PM

Paying more isn't the point tbh. This platform is designed to let you do what you want, which means you need to bring your own batteries, and sometimes the motherboard and storage too, but the CPU is there and its cheap and feature aplenty

rubenfOP•10/29/24, 9:42 PM

I mean some part of our billing service is down for an hour now, and no we just know that a ticket has been raised. It's nice but it's not great. It's cheap, that's for sure, I was hoping it was more than that.

1984 Ford Laser•10/29/24, 9:43 PM

An hour? Thats all? I think you may need to temper your expectations a bit

itsmatteomanf•10/29/24, 9:44 PM

You always need a backup if a system is critical... you need disaster recovery, you need backups.

itsmatteomanf•10/29/24, 9:45 PM

Your billing service is not critical infra, imho, so a downtime is more than fine. Sure, it might not look good, but that's for you to decide how worth that is.

itsmatteomanf•10/29/24, 9:45 PM

If billing is critical you get a secondary (or even a tertiary) system to be switched to as soon as things fail.

itsmatteomanf•10/29/24, 9:45 PM

It's not a if things fail, it's always a when.

1984 Ford Laser•10/29/24, 9:46 PM

Cloudflare will go down at some point, every platform will. You just need to be ready

rubenfOP•10/29/24, 9:46 PM

Yes that's my point, we fortunately are not running critical infra on D1. I do think it's a non-goal for those systems to stay toyish

itsmatteomanf•10/29/24, 9:46 PM

If you were running critical infra, you'd have to have a backup

1984 Ford Laser•10/29/24, 9:47 PM

Also 99.999% of the time, issues experience on the developer platform are directly caused by its use, not the platform itself

rubenfOP•10/29/24, 9:47 PM

Yes and I would still be extremely mad about this

itsmatteomanf•10/29/24, 9:48 PM

Again, it's a pain, but every platform will fail. You don't have to be mad, it's not the platform's fault.

itsmatteomanf•10/29/24, 9:48 PM

It happens, everywhere.

itsmatteomanf•10/29/24, 9:48 PM

You need to be prepared

rubenfOP•10/29/24, 9:49 PM

The whole art of building distributed systems is to make systems that do not fail. You need to be prepared for the impossible sure but you're almost saying this kind of level of service is to be expected on D1

fry69•10/29/24, 9:49 PM

Developers make a typo in a maintenance or migration script and initialize a production db fresh...

itsmatteomanf•10/29/24, 9:50 PM

I'm saying it's expected on every service, no service is 100% guaranteed forever.

Rrubenf The whole art of building distributed systems is to make systems that do not fai...

1984 Ford Laser•10/29/24, 9:51 PM

Windmill has never had unexpected downtime? That lasted longer than an hour? If I was a customer of Windmill I would be mad that you haven't built a backup of the billing service as I pay you to do that

rubenfOP•10/29/24, 9:51 PM

there is a world between no service is 100% guaranteed to this situation is to be expected and the average experience

11984 Ford Laser Windmill has *never* had unexpected downtime? That lasted longer than an hour? I...

rubenfOP•10/29/24, 9:51 PM

correct, Windmill was never down for an hour

1984 Ford Laser•10/29/24, 9:51 PM

but your billing service is?

Rrubenf The whole art of building distributed systems is to make systems that do not fai...

Chaika•10/29/24, 9:52 PM

D1 is still single instance fundamentally, no clue how you would design a "distributed d1" that wouldn't have issues if something went wrong with the data or load. Even if you sharded it or something, you'd have something broken

Rrubenf there is a world between no service is 100% guaranteed to this situation is to b...

Isaac McFadyen•10/29/24, 9:52 PM

I think what we're more trying to say is that downtime is expected for any service. Of course it's not wanted and devs try to make sure downtime never happens, but it inevitably will and because of that, offsite/off service backups are a must.

Rrubenf there is a world between no service is 100% guaranteed to this situation is to b...

itsmatteomanf•10/29/24, 9:52 PM

It's not expected, nor the average experience. It's a single instance being borked, my D1 database is perfectly fine, and it's been so, for months and months at billions of row read a month.

Chaika•10/29/24, 9:53 PM

no one particularly wants to hear they should have had backups after something happened though, espec when it still shouldn't happen

Chaika•10/29/24, 9:53 PM

in a perfect world you would also have software without any bugs though :shrug:

Idk if I agree, the whole point of having multi AZ deployments is that you can trust services like a

Similar Threads

Similar Threads

Similar Threads