Windmill has never had unexpected downtime? That lasted longer than an hour? If I was a customer o

Windmill has never had unexpected downtime? That lasted longer than an hour? If I was a customer of Windmill I would be mad that you haven't built a backup of the billing service as I pay you to do that

rubenf•10/29/24, 9:51 PM

there is a world between no service is 100% guaranteed to this situation is to be expected and the average experience

11984 Ford Laser Windmill has *never* had unexpected downtime? That lasted longer than an hour? I...

rubenf•10/29/24, 9:51 PM

correct, Windmill was never down for an hour

1984 Ford LaserOP•10/29/24, 9:51 PM

but your billing service is?

Rrubenf The whole art of building distributed systems is to make systems that do not fai...

Chaika•10/29/24, 9:52 PM

D1 is still single instance fundamentally, no clue how you would design a "distributed d1" that wouldn't have issues if something went wrong with the data or load. Even if you sharded it or something, you'd have something broken

Rrubenf there is a world between no service is 100% guaranteed to this situation is to b...

Isaac McFadyen•10/29/24, 9:52 PM

I think what we're more trying to say is that downtime is expected for any service. Of course it's not wanted and devs try to make sure downtime never happens, but it inevitably will and because of that, offsite/off service backups are a must.

Rrubenf there is a world between no service is 100% guaranteed to this situation is to b...

itsmatteomanf•10/29/24, 9:52 PM

It's not expected, nor the average experience. It's a single instance being borked, my D1 database is perfectly fine, and it's been so, for months and months at billions of row read a month.

Chaika•10/29/24, 9:53 PM

no one particularly wants to hear they should have had backups after something happened though, espec when it still shouldn't happen

Chaika•10/29/24, 9:53 PM

in a perfect world you would also have software without any bugs though :shrug:

CChaika in a perfect world you would also have software without any bugs though :shrug:

fry69•10/29/24, 9:54 PM

Even if the software is "perfect" people will make mistakes while using it.

Chaika•10/29/24, 9:56 PM

worth explaining if it hasn't already though, green names aren't employees/just volunteers, hence why we can do nothing but escalate and we can do nothing but chat about it

rubenf•10/29/24, 9:56 PM

I understand that, I do not think CF employees would have said that it's laughable to be mad about a downtime of "just an hour"

itsmatteomanf•10/29/24, 9:57 PM

No one said it's laughable

itsmatteomanf•10/29/24, 9:58 PM

Everyone said that it's expected sometimes, and if a service can't (or doesn't want) to have a 1h downtime it needs alternative systems to handle those

itsmatteomanf•10/29/24, 9:58 PM

Issues do happen, either by the devs themselves, by the platform, or any other reason

James•10/29/24, 9:58 PM

agreeing w @rubenf , there's a world of difference between guaranteed SLAs and complete fault tolerance and accepting risk but also expecting your db provider to have some degree of recoverability.
it's in the docs right there, there is distributed consensus built atop the single instance service D1 + DoSQL. The minimum i'd expect is recoverability of data up to the failure point regardless of db corruption. Whether that recoverability is immediate or not is another discussion.

rubenf•10/29/24, 9:59 PM

I just don't get it, yes I understand a single node can fail, I think it's a valid concern that backup since they seem to exist are not usable in any sort of self-service. There is little point saying that "I should just know better and backup myself"

rubenf•10/29/24, 10:00 PM

We do not even need those backups in this present case for the record, i'm surprised they do not exist and it does make me pause at the thought of running actual critical production services on it

James•10/29/24, 10:04 PM

It's absolutely a valid concern and I don't think anyone is arguing that - if you search this chat you'll find reports of these problems intermittently going back since the start of D1. It's wayyyyy less of an occurrence today than it was some time ago, but it does still happen from time to time. Backups are always good, but yes I would entirely agree with you that it's a little wild that a database can fail on Cloudflare so explosively despite all of the marketing they do about global availability, etc. Sadly, there's not a whole lot we can do here except advocate on your behalf and escalate when issues do happen. If you are a biz/ent customer, submitting a ticket about this experience to get escalated is likely the best way to get more eyeballs on it.

James•10/29/24, 10:05 PM

cc @Vy a lot of great feedback and user experience in the messages and discussion above ^

rubenf•10/29/24, 10:06 PM

I think we're in agreement. If it's exceptional then exceptional stuff happen from time to time. I'll see what CF respond and in what timeline. We're likely to share our experience more broadly with other companies to learn from it as I would have liked hearing about those stories when making our infra choices.

JJames agreeing w @rubenf , there's a world of difference between guaranteed SLAs and c...

Chaika•10/29/24, 10:06 PM

fwiw no one has said anything's lost/unrecoverable, still waiting on escalation response, I'd expect the same though

Rrubenf I think we're in agreement. If it's exceptional then exceptional stuff happen fr...

James•10/29/24, 10:07 PM

Agreed, this is exactly what I'd recommend. I would fully anticipate your DB to be restored, but sharing your experience to help others is a great idea. :MeowHeartCloudflare:

1984 Ford LaserOP•10/29/24, 10:08 PM

@rubenf are you able to share here what the eventual issue/resolution is? Def curious to see where it goes

1984 Ford LaserOP•10/29/24, 10:09 PM

Apologies if I came across as attacking or anything, not my intention :thisisfine:

Rrubenf I think we're in agreement. If it's exceptional then exceptional stuff happen fr...

James•10/29/24, 10:22 PM

For reference on usage, here's ~9 months worth of errors of internal, upstream errors for a project using D1: https://i.james.pub/file/2024/10/eafd02a4-02c7-45ba-8cf4-bfe39abbe7d7.png

James•10/29/24, 10:23 PM

1k in the last 7 days https://i.james.pub/file/2024/10/b14ed4b7-a681-457b-92ff-56459d894871.png

rubenf•10/29/24, 10:23 PM

James•10/29/24, 10:26 PM

We're seeing a couple more people reporting similar problems for what it's worth:
https://community.cloudflare.com/t/d1-error-internal-error-while-starting-up-d1-storage-caused-object-to-be-reset/730624?u=cherryjimbo
https://community.cloudflare.com/t/d1-database-error/730656?u=cherryjimbo

We're going to escalate this with higher priority and hopefully get some eyeballs on it ASAP.

Chaika•10/29/24, 10:52 PM

https://www.cloudflarestatus.com/incidents/h5n23gyb4my2

Increase in D1 Errors

naourass•10/30/24, 12:19 AM

Hi there, I'm in a scenario where I update/add data periodically to a local d1 db and need to sync that data progressively with the remote one. Is there some way to connect to the remote db from a node script so I can run the diffing logic (primarily based on last_updatedlast_updated fields) and push the updates from the local db to the remote one, or is

wrangler d1 execute

wrangler d1 execute

the only way currently to run queries on a remote d1 db? Thanks in advance.

Original message was deleted

KaiM•10/30/24, 12:44 AM

Just following up here. With Smart Placement enabled the Worker was running in Brisbane - still very much the opposite side of the world to the D1 database. Any ideas?

Nnaourass Hi there, I'm in a scenario where I update/add data periodically to a local d1 d...

Isaac McFadyen•10/30/24, 12:45 AM

You can use the Cloudflare API, one sec.

Isaac McFadyen•10/30/24, 12:45 AM

https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

sz19•10/30/24, 12:51 AM

The d1 errors are back again in my end

TomK•10/30/24, 3:06 AM

On my end, there have been no recurrences of the D1 errors in the last hour.

TomK•10/30/24, 3:13 AM

Does anyone know how to find the location selected when a D1 database was created? I've looked in both the dash and the wrangler command, but I can't find any relevant information.

TTomK Does anyone know how to find the location selected when a D1 database was create...

Isaac McFadyen•10/30/24, 3:17 AM

Try wrangler d1 info <dbname>wrangler d1 info <dbname>

IIsaac McFadyen Try `wrangler d1 info <dbname>` 😄

TomK•10/30/24, 3:39 AM

There is no location information in the info command.

TomK•10/30/24, 3:40 AM

TomK•10/30/24, 3:40 AM

wrangler 3.83.0

Isaac McFadyen•10/30/24, 3:40 AM

If you do

wrangler d1 list

wrangler d1 list

do you see

production

production

as the version?

Isaac McFadyen•10/30/24, 3:41 AM

There should be a running_in_regionrunning_in_region field right between the size and read query count... strange that you don't see it

FFlikTeoh https://discord.com/channels/595317990191398933/799437470004412476/1298726326265...

Chaika•10/30/24, 3:55 AM

anything more interesting output if you try info with the --json flag after the name?

FFlikTeoh Nope. I had other db that wasn't using location hint though. Those shows its cur...

Chaika•10/30/24, 3:58 AM

What do you mean "Current regions"? D1 only runs in a single region/location, if you don't specify a location hint it just does it off the requestor's location, and it'll stay there forever

FFlikTeoh Yeah. What I mean is that, other db that was created without location hints, sho...

Chaika•10/30/24, 4:01 AM

hmm I can't reproduce that. If you have a second, try making another with the same location hint? Just trying to understand scope, I'd guess it's a weird one-off issue
The --json option is just returning what the database get endpoint responds with + graphql, so it missing is api being confused/not a specific wrangler thing

Windmill has never had unexpected downtime? That lasted longer than an hour? If I was a customer o

Similar Threads

Similar Threads

Similar Threads

Windmill has *never* had unexpected downtime? That lasted longer than an hour? If I was a customer o

Similar Threads

Similar Threads

Similar Threads

Windmill has never had unexpected downtime? That lasted longer than an hour? If I was a customer o