Cloudflare Developers•2y ago

status pages are manual and need to go through people to be approved, seperate from internal inciden

status pages are manual and need to go through people to be approved, seperate from internal incidents

Sébastien Morel•5/29/24, 1:45 AM

ok now all the dbs are gone

SSébastien Morel ok now all the dbs are gone

jLynx•5/29/24, 1:45 AM

rip

jLynx•5/29/24, 1:46 AM

I think this is the first d1 outage we have experienced, so not too bad tbh

Sébastien Morel•5/29/24, 1:46 AM

I agree, they'll fix this I am sure

Koosh•5/29/24, 1:46 AM

we launched something this afternoon - just glad the CEO is asleep while us devs are up at 3am

CChaika status pages are manual and need to go through people to be approved, seperate f...

Daniel Ray•5/29/24, 1:47 AM

This is now on our status page: https://www.cloudflarestatus.com/incidents/3xmdy3cdn85v

Cloudflare D1 Issues

jLynx•5/29/24, 1:47 AM

luckily its almost 2pm here for us

Sébastien Morel•5/29/24, 1:49 AM

kind of critical here, but most of users are in Europe so I guess we have a good 10 hours in front of us

jLynx•5/29/24, 1:49 AM

They are usually pretty fast. sub 30 minutes

jLynx•5/29/24, 1:51 AM

We were frantically looking though all our recent commits to see what we did to break it

Koosh•5/29/24, 1:51 AM

they got about 3 hours until the CEO wakes up - I am going to sleep, please fix before he wakes as out app is down

jLynx•5/29/24, 1:52 AM

no data loss

JjLynx We were frantically looking though all our recent commits to see what we did to ...

Koosh•5/29/24, 1:53 AM

just ran a select query myself in console - then it errored, then bang - db went down straight after - then quadruple checked it wasn't a DROP

DDefenestrator We've had D1_Errors on a few occasions usually only lasting a few mins, we prob ...

YokoWasis•5/29/24, 2:12 AM

but it's already out of beta.

Sébastien Morel•5/29/24, 2:12 AM

"Identified - The issue has been identified and a fix is being implemented." yeah!

YokoWasis•5/29/24, 2:13 AM

Why when it breaks, it always on my country work hours. My phone ringing non stop.

jLynx•5/29/24, 2:14 AM

I would really be interested to understand what caused this issue. @Daniel Ray if someone could provide a writeup after its all done that would be amazing!

jpolstre•5/29/24, 2:22 AM

I was working for more than 2 months on a project with D1, I am disappointed, I feel like I wasted time. When will D1 be used stably? I come from AWS Dynamo, previously FIrebase.

JjLynx I would really be interested to understand what caused this issue. @Daniel Ray i...

Daniel Ray•5/29/24, 2:29 AM

Once all the urgent things have been taken care of, I'll check with engineering to see what can be done.

DDaniel Ray Once all the urgent things have been taken care of, I'll check with engineering ...

jLynx•5/29/24, 2:29 AM

you legend

Jjpolstre I was working for more than 2 months on a project with D1, I am disappointed, I ...

Unsmart•5/29/24, 2:37 AM

D1 can be used stably. Every service has downtime that happens occasionally such as this incident. AWS Dynamo and Firebase have also both had incidents before.

malone•5/29/24, 2:38 AM

For people with urgent D1 needs: I deployed a new version and my D1 started working again

UUnsmart D1 can be used stably. Every service has downtime that happens occasionally such...

YokoWasis•5/29/24, 2:39 AM

I don't know about AWS Dynamo. But I have never experienced downtime using Firestore for 3 years, probably because they have some kind of redundancy / replica in case accident happens.

Unsmart•5/29/24, 2:43 AM

Sounds like you got pretty lucky as they do list quite a bit of stuff happening just in even more recent time

https://status.firebase.google.com/summary

UUnsmart Sounds like you got pretty lucky as they do list quite a bit of stuff happening ...

YokoWasis•5/29/24, 2:45 AM

I am talking about the firestore as a database. Not the firebase service.

Unsmart•5/29/24, 2:46 AM

Still lucky

https://status.cloud.google.com/products/CETSkT92V21G6A1x28me/history

Sébastien Morel•5/29/24, 2:46 AM

incidents happen all the time and they will still happen until chatgpt will control everything. LoL. But for real, the value Cloudlfare provides allow for some incidents once in a while

UUnsmart Still lucky 😛 <https://status.cloud.google.com/products/CETSkT92V21G6A1x28me/hi...

YokoWasis•5/29/24, 2:47 AM

Yeah, I guess I am, because I am in Asia.

ChaikaOP•5/29/24, 2:49 AM

everything eventually has issues, too many moving pieces. What matters more is how they respond & fix it. D1 is def more prone to issues too with it being sql and a single durable object in a single location, rather then something like Firestore which is nosql over many machines and always being replicated. Somewhat just trade offs to nosql vs sql though

ChaikaOP•5/29/24, 2:51 AM

seems that some of the d1 dbs have recovered, at least based off the overview of my acct, like 2/4 or something. I had ~3-4 which never were impacted curiously.

CChaika seems that some of the d1 dbs have recovered, at least based off the overview of...

Kyzer•5/29/24, 2:59 AM

I'm the opposite, only had 1 down, now a 2nd one has gone down in the last couple minutes
Picked the worst two as well

, prod and the one I'm trying to code on today

jLynx•5/29/24, 3:06 AM

oh no @Daniel Ray seems AI gateway is down too now

JjLynx oh no @Daniel Ray seems AI gateway is down too now

Daniel Ray•5/29/24, 3:09 AM

How are you using AI gateway and what errors are you seeing?

Feel free to PM any screenshots so that you don't have to post private information publicly

Is anybody else having issues with AI gateway?

jLynx•5/29/24, 3:09 AM

Let me PM you now