R
Railway10mo ago
Froge

❗Urgent - Deleted fork environment, all database volumes removed from production 🙃

alert were in the process of preparing to test an isolated environment, which was forked. We shut it down after it didn't work as expected, and it appears all our volumes from our databases in our development and production environments were removed... As it's a new environment backups were not enabled, we were actually in the process of setting them up, however we do have a backup from a few days ago. Thee was some important time series information in there, is there a way we can restore that database as we didn't explicitly delete it, so the volume (should, I hope) still exist in your backend and just need re-attaching.
36 Replies
Percy
Percy10mo ago
Project ID: 80d54f52-7f6a-415c-b857-cb6aef417c81
Froge
Froge10mo ago
80d54f52-7f6a-415c-b857-cb6aef417c81
Duchess
Duchess10mo ago
Thread has been flagged to Railway team by @Brody.
Froge
Froge10mo ago
<a:party_railway:812486353853218816> pepelovehearts
Brody
Brody10mo ago
its still Sunday night for a good chunk of the team, please be patient
Froge
Froge10mo ago
No worries, we have created a new database, and restored the backup, we won't be touching the old ones for now. Happy to be patient.
Brody
Brody10mo ago
in the mean time can you provide more clarification? the volumes where deleted off of the service? leaving a database service without a volume?
Froge
Froge10mo ago
Correct!
Brody
Brody10mo ago
mind sending some screenshots
Froge
Froge10mo ago
- All our deploys disappeared, and turned off - Database volumes deleted Steps to cause the issue: - Fork development environment (not production) - We reconfigured some services (so we wouldn't get duplicates on 2 of our listening services) - Then decided we didn't need the fork, removed it - Our monitoring system detected everything was offline - Checked Railway, all deployments gone on everything (even the items we didn't touch in the fork) Steps taken since - Created a new DB, restored backup from 2 days ago (we're still missing 2 days worth of time series data) - Pointed APIs at the new DB
Froge
Froge10mo ago
No description
Brody
Brody10mo ago
that definitely goes against what the docs says
No description
Froge
Froge10mo ago
That's what I thought! 🙃
Brody
Brody10mo ago
can you show me how you made the fork of the dev env?
Froge
Froge10mo ago
Sure It was 100% a fork - I saw the arrow icon below the dev environment, pointing to the fork environment in the environments section in app - When we made a change it said "merge changes" at the bottom
Froge
Froge10mo ago
No description
Brody
Brody10mo ago
I have notified the team I mean I have directly notified the team via chat and the flag
Froge
Froge10mo ago
Thank you ser 🙏 Just as an FYI, my team is in Australia, it has just gone midnight, it may be 6-7 hours before we can respond from now. But would love an update in the morning. Would also love to discuss enterprise plan too! Hey yeah, just checking in?
Brody
Brody10mo ago
theres a good chunk of the team that are still off today
Froge
Froge10mo ago
Right, can we get someone to look? I'm sure a quick check of whether the volume still exists should be relatively quick and simple? That's a good place to start IMO
Brody
Brody10mo ago
this would be likey something jr would deal with, but as far as i know its a holiday for him today
Froge
Froge10mo ago
There's only a single staff member who can check? <:pepe_okbruh:679051001189105702> What does that look like on enterprise?
Brody
Brody10mo ago
enterprise offers custom support, so if you wanted i dont see why you couldn’t get on call support if you wanted
Froge
Froge10mo ago
Noted But that would assume JR isn't the only person who can check something like this, right?
Brody
Brody10mo ago
right, there would be a chain of whos on call
Froge
Froge10mo ago
Okay So we're waiting another day for someone to look at it?
Ray
Ray10mo ago
It deleted production volumes when you deleted the forked env?
Froge
Froge10mo ago
Yep, and all the deployments if the environments stopped
Ray
Ray10mo ago
core and game services, right? and game-2 is something you spun up after?
Froge
Froge10mo ago
Yep, the volumes gone The other services just lost their deployments in production and development Yeah we had a 2 day old backup, which we restored to. We didn't want to add a new volume just in case
Ray
Ray10mo ago
This is 100% on us 😦 Volumes are shared across forks right now; we have a new feature in preview that fixes this (staged changes) What can I do to make this right for you? We can attempt to restore your volumes and the data will be >24hr old depending on when we snapshotted the host
Froge
Froge10mo ago
Yeah okay Unsure why they delete though? Can you get a timestamp of that snapshot?
Ray
Ray10mo ago
Looking into it now. It likely will be just 1d fresher than your backup, tho 😕
Unsure why they delete though?
When you delete a volume in a forked environment, it deletes it in all environments because forks are not 100% isolated copies in the current flow Staged Changes solves that by isolating them within the environment they're in. You can enable that at https://railway.app/account/feature-flags We really should've put some giant warning banner on this current flow, sorry about that I'm gonna credit you 3x your last bill for the inconvenience caused, and will look into the timestamp of our backups for you
Froge
Froge10mo ago
Thank you, we greatly appreciate it! We are a software agency, and have some clients with some decent apps, and we love Railway for not having to think about infra (and that's coming from an ex-infra guy). Would love to discuss enterprise options, we're building everything new in Railway and will be scaling these services!
Ray
Ray10mo ago
Send you the credits!
Would love to discuss enterprise options, we're building everything new in Railway and will be scaling these services!
Can you see the #direct-support channel?
Froge
Froge10mo ago
Yep! Got it 🔥
Want results from more Discord servers?
Add your server