Why no gpu in canada data center today?
My network volume is in ca-mtl-1, there is no any gpu now.
Solution:Jump to solution
Hey y'all, we disable the creation of new pods four days before a maintenance to stop further issues (this was not something I was personally aware of until now otherwise it would have been posted in #🚨｜incidents). However, I talked with the team and you should be able to create new pods again, let me know if you're running into any issues.
20 Replies
Read #🚨｜incidents , its scheduled for maintenance thats why
@haris @Finley it's been more than 4 hours since the outage started. aren't you going to declare an incident and give some updates? looking at the green status on https://uptime.runpod.io, I suspect that your monitoring has not caught this issue.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
@nerdylive Its because RunPod disables the DC before maintenance is about to begin, probably because people don't read and then they log unneccessary support tickets.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I already mentioned this elsewhere, but @fireice being an idiot and giving me a thumbs down already proves my point.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I know this from previous experience, Zeen or someone like that mentioned it.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yes
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
No point in allowing someone to create a pod and have training that runs for days and gets interrupted
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah agreed, RunPod communication is ALWAYS appalling, its about 1% better but still has a LONG way to go
just got an email response from them confirming what @digigoblin says. they are disabling new machine creations.
the morale, as there is no way to clone network volumes (correct me if I'm wrong), you better continuously make backups using https://syncthing.net/ or something like that.
Yep, I guess this is the point of the lack of communication, people need to know when a DC is going to be taken offline for maintenace a few days in advance so that they can start migrating their data to a different DC. When #🚨｜incidents says its only going to be offline for maintenance on Monday, but no new pods can be created 4 days ahead of time, then its a problem because people can't access their data to make alternate arragments. @haris
I mean you should always do backup when you upload data as cloud is basically someone else computer
Solution
Hey y'all, we disable the creation of new pods four days before a maintenance to stop further issues (this was not something I was personally aware of until now otherwise it would have been posted in #🚨｜incidents). However, I talked with the team and you should be able to create new pods again, let me know if you're running into any issues.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yep, as far as I know but I will double check