System fails to boot after deleting non-system btrfs subvolume
Updated bazzite and rebooted, I get this error message. Secure boot is disabled
Solution:
Jump to solution
okay weird, my /etc/fstab seems to contain the /mnt/old_home folder which I deleted also contains the @home-0 subvol i was talking about earlier
Edited:
Solution...
274 Replies
boot previous deployment
None of them work
When I boot into any of them I get a cursor at the edge for a long time, then the bazzite loading screen and then that error message
If I enable secure boot, tells me it needs to load kernel first. I thought all the modules were signed so why would that issue arise?
hmm there is a rare issue that can explain this
What rare issue?
i don't quite understand OSTree
but it has a
/ostree/boot with some symlinks for instances of OSes
links can sometimes get messed up
for some strange reasonLike that jayztwocents issue?
yeah that
we can check
the initrd will still boot
you can still run some commands to fix it
.
Nothing boots up
No access to console, so can't run any commands
yes you can
if nothing booted you wouldn't get this far hmm now that i think of it
when the initrd fails it gives you console access
without a password or anything
There is nothing I can do to gain console access, pressing enter gives me the same error message again. I cannot do anything other than press enter
try editing grub
somebody had a similar problem adding
init=/bin/bash to grub
caused it to fail & drop to the initrd consoleCan I resolve the issue with console access though?
if any of us knew how to use OSTree almost certainly but i'm not that familiar with it so maybe we can i got a few ideas
You can load up the bazzite iso then chroot into your install
Id rather reinstall
But I want to know why this issue would even arise
What could even cause the issue?
filesystem corruption can
Yes but how? I didn't force off, clicked restart, restart acted weird and glitched but it did restart
Why would it not block the restart if there was an issue
plug in a usb stick
mount it
and post the .txt here rdosreport.txt
post what txt and mount what?
oh wait you have to do something else first so you can actually see the logs
sec
Booting to Rescue & Emergency Mode - Bazzite Documentation
Bazzite is a custom image built upon Fedora Atomic Desktops that brings the best of Linux gaming to all of your devices.
use this guide to get past the locked sulogin
okay modify grub entries and get into emergency mode right
yeah
otherwise we cant figure out why it wont boot
after you get a root shell we can figure out why boot is failing
Okay
I think I'm in
Says bash-5.2#
why is this so annoying anyways
why does a system without a root account still require a root password(sort of) to access the system's shell
i thought fedora had a fix
at some point
good
you're in
What do I do now
plug in a usb stick
mount it using shell
or maybe just take screenshots
journalctl -b 0Yea ss better I guess
will show current boot
photo with your phone*
screenshots aint gonna work
sorry
do
journalctl -b 0 hold spacebar till the end and find where there's an errorHmm 66blines
Okay wait lemme find the error
then just make a picture and post it
Are the error in a different colour or font?
Why are the logs showing sep 22
Is that relevant?
Its sep 21 for me
all of this is normal ish
real error obably much further down
just about there
looks good so far
no very bad error from what i could tell
Welp I hope there isn't personal info in the logs
these logs are from the initrd
so it's mostly hardware info
This is from sep 21
w we're switching to the real system
this seems like not that big of an issue
The journal logs are split into sep 21 and 22, why would that happen? Still sep 21 here
not entirely sure but have seen it before
i think it's not the symlink issue it wouldn't boot this far
When I installed bazzite, I had an old btrfs home subvolume and it gave me an error,
So I mounted the old one as /mnt/old_home with name @home and made a new btrfs subvolume @home-0 and mounted as /var/home
After system was installed I unmounted /mnt/old_home and then deleted the btrfs subvolume. Can that be an issue?
I've had some weird issues with btrfs like that before
logs should answer that
eventually
What part through? Can I not grep it and check?
you should be able to
you just pipe into grep
But search what
not entirely sure but this should help
journalctl -b 0 -p err
this only lists errors
That was fixed last month
oh cool
was a real annoying problem
hmm this looks like a successful boot to me
try
journalct -b -1 -p err
this shows you the previous boot attempt
errorsCould those ACL errors cause mounting of root to fail?
root at least the main root partition was mounted correctly
maybe not the OSTree root
but certainly the partition that's on
actually now that i think of it
both are mounted
lemme find the screenshot that proves that
see where it says switching root
if it couldn't be mounted the switch couldn't happem
Does show switching root
so the system is found & somewhat booted but fails later on
This was from a different boot ( I tried rebooting again)
Journalctl didn't keep logs from the first boot (which the photo is from)
Welp I guess I'll just reinstall everything and hope that it doesn't have the same error next time
Hey guys
I reinstalled and I did the exact same thing and I got the exact same issue
I deleted a subvolume called @home-0 and after rebooting I get the same error
Can someone look into this and check if this is supposed to happen?
Everything worked before I deleted that subvolume
I do not understand why that would lead to an error?
why are you deleting home?
I did not delete @home
I deleted a subvolume called @home-0 which was from a previous bazzite install, I wanted to mount it seperately and transfer my files from that subvolume to my actual @home subvolume after installing bazzite
I couldn't mount my original home volume because it gave me a warning and asked me to create a new filesystem
right but from the looks of things you deleted the home folder that was being used
Nope
I had two subvolume @home-0 and @home
Deleted @home-0 only
I do not understand why the system kills itself when I delete a subvolume not being used
Do you dual boot with Windows, by chance? I just saw a youtube video this morning with this issue. The thinking was that a Windows update on the shared boot drive overwrote some files for your Bazzite install
Not applicable here
gotcha
Does the name @home-0 just make bazzite go crazy?
the filesystem doesn't care what it's named for the most part
it cares where it's mounted
Yes so why does this happen
Can someone try doing a fresh install and checking
Idk if it's only my system
because @home-0 was mounted at /var/home probably
or was it usr
No
I did not
@home-0 was mounted at /mnt/old_home
And I made a new subvolume called @home and mounted as /var/home
And then bazzite worked, no issues, rebooted and all
Went back in, transferred files from /mnt/old_home to my current home directory (all my personal files, no config files)
Then unmounted /mnt/old_home then deleted subvolume @home-0 using btrfs snapper
And then rebooted and I get the same message as the image I first posted
Second time this has happened
did the uuid of normal home stay the same?
I did not check after I deleted @home-0 but I did do echo $HOME and it pointed to @home
So I don't think I messed with my actual home directory
Why would anything else matter if I deleted unrelated subvolume called @home-0, system is expected to work fine unless this is an issue with btrfs-assistant
I deleted the subvolume using btrfs-assistant
Sorry called it btrfs-snapper I meant btrfs-assistant
This was with the first time I had the issue
did you do it with an external operating system
What do you mean?
did you delete the subvolume while running on the volume
Nope, I deleted the subvolume while running bazzite, btrfs subvolume can be removed while online no?
im not sure if you should though
Yes but why does the option exist then
They should remove it from btrfs snapper tbrh
for the people that know what they're doing
I did not know, I was under the assumption that it could be removed online
and might have a reason to do it
So this is an issue with btrfs and not bazzite then?
btrfs assistant is an advanced tool
maybe
This issue seems to be easily reproducible though, maybe some specific quirk with my system, guess someone else has to try and check it out
Is this not a bug report techicnally?
Where do I submit bug reports
the github
but you'd have to specifically say what issue you're having as in a specific error relating specifically to btrfs that's specifically not something you caused because generally people should not be touching the subvolumes
like there's a difference between an actual bug and a misconfiguration
Is this not an actual bug?
while it could be a bug, you used btrfs assistant which is kinda advanced
and will force whatever you tell it to
Nothing advanced about clicking on the subvolume and clicking delete
if you used gnome disks it'd be a different story
No errors popped up too because I unmounted /mnt/old_home before deleting
right you unmounted it
did you restart after unmounting it
nope only after deleting the subvolume
Why would I need to restart after unmounting a file directory
you can unplug a drive from a live system and it might run for a couple seconds before it tries to access it
you'd never get a critical error if it didn't try to access something important
So this is btrfs not stable issue probably?
no
you messed with the filesystem, and the filesystem borked, simple as
To mess with the filesystem I would have had to do much more
I simply unmounted the folder and then hit delete in btrfs assistant
No other commands were run
This is what is expected of the filesystem to work
you deleted a subvolume, that is messing with the filesystem
Yes but I didn't do something not recommended
I did what was available
you used a tool for advanced users
im sorry but we can't just bar tools because they got used improperly
unless it's kde disk managment
which got barred cuz it broke things
What did I do improperly?
now we have gnome disks
you tell me
do you know enough about filesystems and btrfs to know what might have happened?
That is exactly why I am here
yeah and why I'd recommend to not do advanced configuration like that
Why does that happen? The expected behaviour is for unused subvolumes to be deleted and the system works because no critical system components were not touched
until you do learn about it
The expected behaviour is for the system to run considering the exact steps I took
did gnome disks allow you to delete the subvolume while live?
Gnome disks does not display subvolumes?
What
Please check
you're right
What I did was delete an unrelated subvolume using btrfs assistant.
The fact that it's an advanced tool or noob tool has absolutely nothing to do with what happened to the system. The expected behaviour was the system to continue functioning because that subvolume was not being used
If this is a concern then the bazzite devs must remove btrfs assistant because the bazzite os is considered to be a mostly noob friendly experience akin to linux mint or atleast maybe mention quirks with btrfs assistant and recomend using it only for advanced users
please create an issue on hmm
the wiki i think?
since this is a documentation issue
as for btrfs assistant, it exists because gnome disks can't do much with btrfs, very unfortunate
🤦♂️
And yet I get berated for using btrfs-assistant
So much for a help server
sorry if it came across that way
It did come across that way.
if it makes you feel better we did get an outcome 👍
Outcome as in?
train go boom
Outcome as in there is solution / explanation to what is going on?
uhhh
idk my brain stopped working
this is your problem
your partition is screwed
need to figure out why
if your install dies during switch root its 99.999999% likely your btrfs partition
first make sure to check if the filesystem itself isn't just dead
with a live iso
btrfs check --check-data-csum /dev/<yourbtrfspartition>(also keep in mind a lot of us are fellow users trying to be helpful not anyone paid)
It's not about the help, at the very least I don't want someone to be going on "This is a advanced user thing, whatever you did was your problem"
I clearly explained what I did, I mentioned what subvolumes I made and what I did to delete a subvolume , instead of "yea maybe the issue lies with you deleting this subvolume and it screws with btrfs" or whatever the reason might be, the entire explanation gets ignored and I get something entirely unrelated and condescending
Okay so mount my entire btrfs volume and then run this command. Cool. Ill let you know
Probably because I deleted some unrelated subvolume while I was online. Either btrfs-assistant or btrfs itself is buggy asf
Nope no errors
Also all the critical subvolumes are intact, I didn't delete the wrong one
I ran into a similar-ish issue with CachyOS on btrfs from a firmware update. Completely bricked the install and chrooting couldn't fix the issue. Never figured out exactly why it happened, but the boot partition couldn't mount the OS partition. It claimed it couldn't find the UUID, but when I escaped to the console and ran diagnostics, the UUIDs were all correct.
Did you use btrfs assistant or delete some subvolume?
No
It was just a firmware update in my case
But the end result seems similar to yours
I see, seems btrfs is kind of finnicky
I deleted a subvolume and now it goes into emergency mode without console
Lots of users have had this issue too albeit a bit different
On the forums
As a work around in your case, do you have a live boot drive and a spare USB drive you can temporarily place your home drive files onto? If you can do that, then you would be able to wipe the old home partition during the Bazzite install and not have to delete it after the fact. You could then just mount the USB drive temporarily to transfer your files.
I know it's not an ideal solution since the danger still exists for the sub volume deletion, but you would at least be able to avoid it.
Im fine with reinstalling that's not really a problem, I'm trying to figure out why this issue happens in the first place. Twice this has happened after deleting an unrelated subvolume
I'm kind of wondering if the same issue happens on other distros doing the same operation with btrfs
Solution
okay weird, my /etc/fstab seems to contain the /mnt/old_home folder which I deleted also contains the @home-0 subvol i was talking about earlier
Edited:
Solution
Tl:Dr:
Deleting btrfs subvolume through btrfs-assistant did not remove entries from fstab and caused bazzite to not be able to boot.
Had to comment out deleted subvolume entries in fstab
If it did, that would indicate the bug is in btrfs and not Fedora/Atomic
can any devs help and check if this might have been the issue?
That would probably do it since you don't have a
nofail argument on the sub volumethis is a weird quirk of the anaconda installer then
Wait
i cannot delete a non critical subvolume then
Also, it's listed before your /var and /home get mounted, so that could, and I believe fstab loads in order
Your old home and var have hte same UUID
why would anaconda even do this
i have 0 clue what is going on here rn its upto the devs or someone knowldegable rn to figure this out
i just deleted a subvolume @home-0
well it's a pretty garbage
The UUID is the same for the entries, but I think the
@subvol arg is supposed to filter. You could always change to partition UUIDs instead to be more explicit, though.yes but is that expected behaviour?
also yeah trying to mount something other tham
/ without nofail+ there shouldve been a no fail argument for my @home-0 because it isnt a critical subvolume
I could see fstab not being updated by the tool you used to be expected behavior.
will definitely cause this issu
In any case
i used btrfs-assistant
Comment out, reboot, test
is btrfs-assistant the issue here?
its always failed me
wait since the fstab is read only could it mean that btrfs assistant couldnt write to fstab?
That's likely
damn
so actual serious issue then
i noticed/said it swched root fine
But I'm not sure btrfs assistant would even write to fstab.
yes but i deleted the subvolume from there, so it should have written to fstab no
but this case it cant because read only system
i have no clue, i am super new to this
fstab isn't really read only
/etc can be modified just fineBtrfs assistant is a disk management utility, not a boot management utility. Fstab is for booting. I wouldn't be surprised if it didn't write to fstab at all.
then why wasnt it removed from fstab
i deleted that subvolume
have to call that a bug myself
welp
comment the @home-0 and then try booting?
wait do i need to do anything else here? any information that maintainers might require if this is an actual issue?
@Kasher_CS
should be bootable now
do i need to obtain any other info or is this relevant enough for the devs to do something
Okay ill just comment out that line and check if booting works
donno what else could be useful
I'm not familiar with btrfs-assistant, but the Arch docs mention having to manually modify fstab for pretty much all the other btrfs utilities, so I wouldn't be surprised if it's "working as intended" and it's just one of those footguns once you dive deep enough into the Linux utilities.
is this right?
also must i also make changes in the other hash too?
Yep, that looks good to me
im currently in hash 0\
must i change hash 1 too?
will it not cause some error due to mismatch?
You would probably want to change both in case you roll back.
okay cool
also what is 0.origin
should i change that too?
within there
there are 3 hashes
.0 .0..origin and .1
I'm not super familiar with the inner workings of ostree, but I would imagine the 0.origin is the one that all the others are layered over, so you would probably want to modify that one as well.
okay cool
ill change all then
origin is a file that has something to do with sources
there is a orgin for 1 and 0 too
so change the one in white or blue
like where the system comes from
so where do i change
all 4 of them?
the ones in white
or the ones in blue
files are in white folders are in blue
origin is just an info file i think
oh my bad
will the file have the new data then?
each of the folders contain a system
which is which i'm not entirely sure
i mean i'm not sure which is current/previous
current is 0
i think
previous is 1
but we only really need 1 to work
I think as long as you fix each of the files that are actually an fstab file to not include your old home sub volume, you should be good going forward and backward through your ostree commits.
but i commented both in 0 and 1
should i try rebooting now and check
corss your fingers guys
I would try rebooting
okay imma reboot now
I just close the console and stuff no everything would've been written if I pressed ctrl + o in nanao?
After commenting that is
yes ctrl + o is save
yes
YES
BOOTS UP
FINALLY
Dayum
Btrfs assistant was the problem all along
Also will new boots have this fstab?
Will I have to change?
I guess someone should tell the devs
You should be good
Of btrfs assistant or bazzite I have no clue who
can't count the number of times i messed up fstab
Same
Btrfs assistant should've automatically made this entry
It didn't
I don't think it should have
Then who should have
The delete button is there, it should've removed that subvolume entry from fstab
Why would you place a delete button and then it bricks your system
It doesn't know the drive is in fstab
KDE Partition manager can do mountpoints
Why does the delete button exist then? Also why haven't more people run into this issue
Do they not delete subvolumes?
also yeah managing fstab
isn't really it's main thing
Then what is supposed to make the change?
You, the user
then they should probably remove that delete button for subvolumes from btrfs -assistant
Very misleading
it should definitely not let you delete stuff that's mounted
well technically not mounted
It doesn't, I unmounted and deleted
He unmounted before deleting
but required to be mounted
And it didn't make a fstab entry
But I don't understand why btrfs-assistant doesn't deal with fstab and remove the subvolume when I click the delete button
still allowing the user to do this just seems dumb
That should be it's functionality no?
The thing is that in Linux, disk partition management and fstab are two totally different concerns and this is one of those gotchas where when you don't know, you shoot yourself in the foot.
Or why even have that in the first place
Someone just tell the bazzite devs to either remove btrfs assistant or display a warning that fstab entries have to be manually made
Yes, I think the warning would be a great idea
yes but it shouldn't be separate when it can't be
in this case it couldn't be separated
Or maybe anaconda must add a nofail for non critical subvolumes
editing partition tables is one thing
editing required partitions is another
it should definitely have some safegards
Wait but this isn't partition management, this is subvolume management. Btrfs assistant fails with the one thing it's supposed to do
Are there other disk utilities that modify fstab automatically for you? I'm honestly not familiar with them. All the cli utilities I've ever used let you shoot yourself in the foot.
It doesn't deal with partitions that is the job of gnome disks
This is completely btrfs assistants fault for not modifying fstab entries when it deals with subvolumes
Or someone else's idk
well not automatically
but there are options to modify fstab in KDE Partition Manager
Ah okay
as in you can assign/remove mountpoints of partitions easily
I see, so that utility at least provides a GUI wrapper over editing fstab at least a little bit
still doesn't mark things as nofail by default which is kind of a footgun if you don't know about it
there's a checkbox to mark as nofail though as i recall
Yeah, I honestly don't know why that's not default for any non-boot, non-/ mount point.
is btrfs-assistant a advanced user thing?
why include within bazzite on default install if it is
also why shouldnt btrfs-assistant not write to /etc/fstab, it is writable and its job when deleting is to remove those entries
if you give me a delete button you should not crash the system because you are unsure if you should write to something
Ill just post a message in #🎮bazzite-dev and hope they see it
Btrfs Assistant is a file system utility. Sub volumes are a feature of btrfs. You can pass arguments from fstab to btrfs to specify a sub volume to mount. The fact that your OS failed to load means the sub volume was successfully deleted and was no longer found.
yes but the filesystem utility must tell fstab that this subvolume does not exist and hence remove it from fstab?
or were the devs expecting that if a subvolume is being deletd it probably is not imporant and hence the installer wouldve already added nofail to it in fstab
I believe they expected anyone attempting to mess around with sub volumes would be an advanced user, otherwise they probably wouldn't be messing with them.
Fstab is not part of the file system. The file system is a special part of a disk.
then why make a gui
I don't know
+ this should not be installed on default bazzite if btrfs-assistant is a advanced user tool
I agree there
The footguns should be opt-in only, haha
+ i dont understand how this isnt a more common issue
wouldnt people look at subvolumes and then want to make one for games or photos or whatever
and then one day decide they no longer need it and delete it
I think the problem may lie in the fact that the initial sub volume was created during installation and because it was attached to your home directory, was not allowed to be
nofailno it was not no fail, nofail means it can boot even if its not detected no?
despite being very optional
Nofail means it will not cause a failure if not found, yes
no i manualy created the subvolumes
i couldnt just mount my old home subvolume as /var/home because it gave me a warning
After installation?
of course a DE won't start without somewhere to put
files
but you can still log in to TTY
so i had to make a new one and then mount my old one as /mnt/old_home, then after installing transfer my files to my actual home directory and then delete the old subvolume
without a home
i do not know why that warning pops up
When did you create the sub volumes?
told me to make a new filesystem for /var/home
the @home-0 was a home subvolume from a previous install of fedora i had
it was my old home subvolume
i thought i could just mount it as my new home subvolume in bazzite
ah the installers do that
but i couldnt so i thought ill make a new subvolume for bazzite home and then transfer my files
the point of being able to keep /home was that so distrohopper can distrohop without losing personal files
So this is a Bazzite installer issue for not defaulting a non-critical mount point as
nofail or at least not giving you the option to select thatyep i guess
just mention that in #🎮bazzite-dev
+ i do not think new users would know what no-fail and stuff means
I do not have access to that channel, I might need to select some other server roles for that
at least anaconda does ask you to re format the filesystem
so idk probably non crtitical subvolumes should be marked as no fail by default?
calamares defaults to keeping the old FS
I think anything that isn't required for Linux to boot should be nofail
That's, of course, my own opinion
bot quite sure why it ain't
fun thing i heard
syystemd can auto generate a basic fstab
based on partition GUIDs
by that i mean
you know how cfdisk lets you mark GPT partitions with a purpose
like Linux Root Partition
Oh yeah
& Linux
/homeKind of similar to tags but less options and stored a little different
well systemd can generate a fstab at boot based on that
Oh interesting
Like, if the actual fstab fails it can default back to a generated one from those purpose markers?
Or you tell systemd to overwrite your fstab with a generated one?
i think it's only if you don't have one
Ahh, okay
also will the new installs have the same fstab? or does it build upon the origin file?
technically systemd generates
.mount units fstab
if there is none .mount units are generated from the GUIDs
on reinstalls the system which includes generating the fstab
on updates it does an install of a new system but it preserves /etc
stuff
including fstab