Cloudflare Developers•2y ago

We were frantically looking though all our recent commits to see what we did to break it 😂

We were frantically looking though all our recent commits to see what we did to break it

Koosh•5/29/24, 1:51 AM

they got about 3 hours until the CEO wakes up - I am going to sleep, please fix before he wakes as out app is down

jLynxOP•5/29/24, 1:52 AM

no data loss

JjLynx We were frantically looking though all our recent commits to see what we did to ...

Koosh•5/29/24, 1:53 AM

just ran a select query myself in console - then it errored, then bang - db went down straight after - then quadruple checked it wasn't a DROP

DDefenestrator We've had D1_Errors on a few occasions usually only lasting a few mins, we prob ...

YokoWasis•5/29/24, 2:12 AM

but it's already out of beta.

Sébastien Morel•5/29/24, 2:12 AM

"Identified - The issue has been identified and a fix is being implemented." yeah!

YokoWasis•5/29/24, 2:13 AM

Why when it breaks, it always on my country work hours. My phone ringing non stop.

jLynxOP•5/29/24, 2:14 AM

I would really be interested to understand what caused this issue. @Daniel Ray if someone could provide a writeup after its all done that would be amazing!

jpolstre•5/29/24, 2:22 AM

I was working for more than 2 months on a project with D1, I am disappointed, I feel like I wasted time. When will D1 be used stably? I come from AWS Dynamo, previously FIrebase.

JjLynx I would really be interested to understand what caused this issue. @Daniel Ray i...

Daniel Ray•5/29/24, 2:29 AM

Once all the urgent things have been taken care of, I'll check with engineering to see what can be done.

DDaniel Ray Once all the urgent things have been taken care of, I'll check with engineering ...

jLynxOP•5/29/24, 2:29 AM

you legend

Jjpolstre I was working for more than 2 months on a project with D1, I am disappointed, I ...

Unsmart•5/29/24, 2:37 AM

D1 can be used stably. Every service has downtime that happens occasionally such as this incident. AWS Dynamo and Firebase have also both had incidents before.

malone•5/29/24, 2:38 AM

For people with urgent D1 needs: I deployed a new version and my D1 started working again

UUnsmart D1 can be used stably. Every service has downtime that happens occasionally such...

YokoWasis•5/29/24, 2:39 AM

I don't know about AWS Dynamo. But I have never experienced downtime using Firestore for 3 years, probably because they have some kind of redundancy / replica in case accident happens.

Unsmart•5/29/24, 2:43 AM

Sounds like you got pretty lucky as they do list quite a bit of stuff happening just in even more recent time

https://status.firebase.google.com/summary

UUnsmart Sounds like you got pretty lucky as they do list quite a bit of stuff happening ...

YokoWasis•5/29/24, 2:45 AM

I am talking about the firestore as a database. Not the firebase service.

Unsmart•5/29/24, 2:46 AM

Still lucky

https://status.cloud.google.com/products/CETSkT92V21G6A1x28me/history

Sébastien Morel•5/29/24, 2:46 AM

incidents happen all the time and they will still happen until chatgpt will control everything. LoL. But for real, the value Cloudlfare provides allow for some incidents once in a while

UUnsmart Still lucky 😛 <https://status.cloud.google.com/products/CETSkT92V21G6A1x28me/hi...

YokoWasis•5/29/24, 2:47 AM

Yeah, I guess I am, because I am in Asia.

Chaika•5/29/24, 2:49 AM

everything eventually has issues, too many moving pieces. What matters more is how they respond & fix it. D1 is def more prone to issues too with it being sql and a single durable object in a single location, rather then something like Firestore which is nosql over many machines and always being replicated. Somewhat just trade offs to nosql vs sql though

Chaika•5/29/24, 2:51 AM

seems that some of the d1 dbs have recovered, at least based off the overview of my acct, like 2/4 or something. I had ~3-4 which never were impacted curiously.

CChaika seems that some of the d1 dbs have recovered, at least based off the overview of...

Kyzer•5/29/24, 2:59 AM

I'm the opposite, only had 1 down, now a 2nd one has gone down in the last couple minutes
Picked the worst two as well

, prod and the one I'm trying to code on today

jLynxOP•5/29/24, 3:06 AM

oh no @Daniel Ray seems AI gateway is down too now

JjLynx oh no @Daniel Ray seems AI gateway is down too now

Daniel Ray•5/29/24, 3:09 AM

How are you using AI gateway and what errors are you seeing?

Feel free to PM any screenshots so that you don't have to post private information publicly

Is anybody else having issues with AI gateway?

jLynxOP•5/29/24, 3:09 AM

Let me PM you now

Bọc Link VN•5/29/24, 3:10 AM

Is there any update on this? our servers are down

Walshy•5/29/24, 3:11 AM

https://www.cloudflarestatus.com/incidents/3xmdy3cdn85v

Cloudflare D1 Issues

Walshy•5/29/24, 3:11 AM

is the place to watch for updates

Bọc Link VN•5/29/24, 3:11 AM

thanks!

DDefenestrator Yay, D1 seems back for me!

Bọc Link VN•5/29/24, 3:15 AM

but not for me, still loading when access to D1 dashboard

Sébastien Morel•5/29/24, 3:25 AM

1/4 here

Daniel Ray•5/29/24, 3:26 AM

Status had been updated to include AI Gateway as well as D1.
https://www.cloudflarestatus.com/incidents/3xmdy3cdn85v

Cloudflare D1 Issues

jLynxOP•5/29/24, 3:27 AM

I'm sure they will let us know once its all back up and fixed

Original message was deleted

Chaika•5/29/24, 3:27 AM

Location was my first theory but it seems to have nothing to do with it

Bọc Link VN•5/29/24, 3:28 AM

Is there any alternative option for database if we cant't use D1?

Chaika•5/29/24, 3:28 AM

some down in east us, west us, west europe, etc

BBọc Link VN Is there any alternative option for database if we cant't use D1?

Chaika•5/29/24, 3:28 AM

at Cloudflare? It depends what you're looking for

Bọc Link VN•5/29/24, 3:28 AM

yes at cloudflare

Chaika•5/29/24, 3:28 AM

https://developers.cloudflare.com/workers/platform/storage-options/

Cloudflare Docs

Choosing a data or storage product. · Cloudflare Workers docs

Storage and database options available on Cloudflare's developer platform.

Chaika•5/29/24, 3:29 AM

ps. It's not like D1 is going to be down forever, give it a few hours max would be my prediction, hopefully way sooner*
prediction based entirely off hope and prior events because I have absolutely no more insight then you on the issue itself, usually this stuff is pretty quick though in the past with other products that are GA, it's going to depend on the issue too of course but when you've got a GA product having critical issues eventually you'll get enough eyes on it to fix anything

BBọc Link VN Is there any alternative option for database if we cant't use D1?

jLynxOP•5/29/24, 3:29 AM

Its not going to be down forever if thats what you are worried about. I dont suspect it will be too much longer, maybe another 30 minutes or so

Bọc Link VN•5/29/24, 3:30 AM

yes, D1 is fast and working well in cloudflare ecosystem. I use D1 with worker and it working fine for me

Bọc Link VN•5/29/24, 3:31 AM

I know the issue can come anytime . but from technical view I just want to know is there any backup option if D1 is down. maybe Azure SQL database? but dont have option to connect with

BBọc Link VN I know the issue can come anytime . but from technical view I just want to know ...

Chaika•5/29/24, 3:33 AM

For more apps, it's simply not going to be worth engineering anything like that, to be frank.

Chaika•5/29/24, 3:34 AM

too many potential issues with syncing stuff between dbs, finding stuff that works between them (ex: Azure SQL Database is Microsoft's SQL Server, and D1 is SQlite, pretty different), etc

Original message was deleted

Bọc Link VN•5/29/24, 3:36 AM

Maybe we need to review the system design, but it is possible to down one service and the entire system down.

We were frantically looking though all our recent commits to see what we did to break it 😂

Similar Threads

Similar Threads

Similar Threads