Emergency mode after reboot
I was trying to get a design tablet to work following instructions from ublue community and other discord posts here. I ran
rpm-ostree kargs --append=modprobe.blacklist=hid_uclogic to try fix a design tablet issue. On the subsequent boot the tablet actually worked, but then I found that I could not rpm or ujust as they were failing (didn't make a note unfortunately). So I rebooted, I was able to ujust update fine, then rebooted again and its in emergency mode. Any advice appreciated.
I believe all the errors (in red) in my journalclt line up with the poster here - https://discord.com/channels/1072614816579063828/1396940021667467384
I have followed the pinned comment relating to btrfs errors (and saw no mention of BTRFS in red prior or post instructions) - gparted cleaned, reboot and still having the issues with all grub bazzite images.
Emergency mode output on proceeding is - 'Failed to connect to the system scope bus via local transport: No such file or directory'.
ostree-prepare-root[1193]: ostree- prepare-root: couldn't find specified ostree root - - then the directory its looking in
I'm a little over my head, is this related to the command I ran, or something else? @biebel Thank you for your assistance so far.
(I'll be back tomorrow), as is mentioned in the linked thread, there was a suggestion to keep the home directory while merging new boot. If that is the only option open to me, how do I go about doing this (I am dual booted on single nvme currently)131 Replies
try first
yes look here please
thank you.
Ill try now, but I don't think I can rpm from emergency mode when I tried.
are you able to get into the system at all?
I have a bootable iso too (if that helps)
No, none of the images work
same emergency mode, and the b41 are completely borked
or you are stuck in emergency mode
b41?
yes, completely stuck in emergency mode. In grub I get 2x 42 options, both lead to emergency mode, and 2 x 41 option, that seem broken in another way
I followed the pinned post with parted, it seemed to complete successfully - though I followed it as good practice, my systemctl didnt mention BTRFS in any errors.
wait why do you have 4 boot options?
did you upgrade to 42 from 41 some time ago
No idea, it just showed up one deay early on (5 or 6 months ago).
No, I have always been on default branch
I haven't rebased from default bazzite.
ah no that's normal then
At some point early I went from having 2 options in grub to having 4 (actually its 5 as w10 is on the same drive - that I am using currently).
cause 42 only dropped at may iirc
so if you used bazzite before then you would've updated from 41 to 42
I don't recall them specifically always being 42, but that I went from having 1 on first boot - to two (when it set up a rollback), then at some point in those first weeks it became 4 in the grub.
you could try and boot the other 42 from the UEFI
But has never caused an issue, a little reddit cruising indicated other users had this too and were not fussed, so I left it.
yeah that is a bug that should've been fixed in 42
How do I do this, in BIOS I have one boot option, and it is UEFI - my drive. Which opens grub
hmm
ill brb
usually on a bazzite install you would two fedora entries
well
bazzite entries
This is BIOS, in was never certain why I ended up with two entries like this for fedora, again assumed it was fine as its been working for months. The disabled fedora never actually reached grub. The remaining entry was relied upon
I have w11 on a secondary ssd that is disabled from boot
This is what I would get if I booted the disabled boot option
the two entries are normal
can you select the other fedora entry in the top box?
or just manually boot into it with the boot menu
the UEFI boot menu that is
not grub
Whether I select it via F8, or from BIOS, it leads to grub and the outcome is the same as in all options go to grub menu. I can't go straight to fedora. Should I try something else?
for both fedora entries?
Here is parted partition layout
The one I have normally as the only boot option, when selected goes to grub. The other fedora which is normally disabled in bios goes to the screenshot above, I'm not sure what that grub explainer page is.
pretty sure that style of UEFI menu selects the boot order no?
you should either boot manually with the boot menu, or select the other fedora entry as the 1st in boot order
Yes it typically selects. the boot order that I have been used to. Since following the same drive instruction for dual boot. When I select the fedora option from BIOS it goes to my grub menu, with windows on it.
apologies for the grammar and flow, I'm not used to mobile keyboard / discord.
In that case I assume the bazzite 41 entries also doesn't work?
I am not sure if clarification is needed, but - the BIOS menu, I can select the boot order and enable or disable devices. I have learned the following:
Windows 11 drive (disabled, if I need it enable it and select it via the motherboard boot menu, as it is not represented in GRUB).
Windows 10 partition on same drive as bazzite (disabled, it is represented in GRUB menu that is offered when booting the Fedora option from the BIOS order.
Fedora (1 of 2) Launches grub selection - with the 4 bazzite options and windows 10, historically has worked but as of yesterday both 42 go to emergency mode
Fedora (2 of 2) goes to the grub explainer menu, I never understood it, but it was no use when 1 of 2 gave me the grub I was looking for.
So typically I only have Fedora 1 of 2 - boot option in BIOS actually turned on, all other options are disabled as over 99% of my time is in Bazzite.
Unfortunately not, error pic to follow.
In the meantime - the error pics here, I have the same journalctl errors (with grub options 1 and 2 - both b42 options) - https://discord.com/channels/1072614816579063828/1396940021667467384/1396940021667467384 if this helps.
can you give me the journalctl output?
a photo of the screen is fine
This is either of the baz41 options, then it returns to grub selection
Any journalctl command in particular? Just the red bits? Its many pages long
Ooo, I've just noticed. I seem to be on a specific bazzite, dated may
you are in now?
No sorry, i was recalling reading a date for bazzite on the first line of journalctl. I'm still dead in the water
can you scroll to any red bit and take a photo
Really slow uploading, apologies
Here is emergency mode btw
ok yeah you probably have to reinstall then
what leads you to the conclusion out of interest? Trying to learn from this. Do we feel it is due to the karg that I ran (as explained in OP), or generic wonkiness of another sort?
Can you check if the uuids of the bazzite partitions are the same in fstab as they are on those drives? Only fatal error I see is it not finding ostree root
by those drives - where am I determining this to compare against the fstab?
lsblk in emergency mode
Command not found
I also can't nano my fstab from here I assume
Blkid works, Pic incoming
I kinda need to do something else at the moment, sorry that I can't help you til the end
I honestly thought this would be just a rpm ostree rollback at the start :p
You have helped so much. Thank you
k, now let's try to dig out the fstab to see if it corresponds
what do you get from ls?
Ls on its own returns nothing
And I cannot hit my trusty - sudo nano etc/fstab (command not found) (from servers, never touched it in bazzite)
:/ might be way easier to do from the gparted environment
Never done this on an atomic linux and I'm not sure how the magic works, but we can't make it worse
What am in running in gparted?
Find fstab and see the right uuids are in there for /boot and /var mostly
Also a good environment to backup files for a potential reinstall
I was actually going to ask if we get to that point, its its possible to install (from iso) over the current bazzite without losing the home folder, as mentioned in the linked chat. You have experience in this? In the meantime I'm looking for uuids
Not sure that that is a good idea. The safest route, if you have enough free space is to make a backup of your current home folder and import as needed after a reinstall.
I would just delete p6 and 7, or any others (when reinstalling without disturbing w10 on the drive?
Yes and p5 and follow the manual partitioning guide
So, I have mounted the partition to mnt, but, boot and var are empty. My user files are there, but anything OS related, I see the top level directory then empty
do you have /etc ? that's where fstab is
/boot will be on the ext4 partition. It's part of the magic I don't quite understand in atomic distros
No etc. I dont understand actually why mounting the partition results in 3 options, as shown in Ls of mnt above. Pics incoming
It's the btrfs subvolumes
02 just has my user folder in it with all files seemingly present
Understood (the explanation, not how it works). I'm used to etc being there, i assume it is there when using the OS, but not visible in parted
what's under roothome? Might be a better idea to have someone here that's not also exploring this from the outside for the first time :/
Is there a magic lamp with which we can summon such 🙂 I have to repeat, I really appreciate both of your help and willingness to explore. I felt with it being atomic it really limited my ability to scout answers through search engines as I normally would. As I'm not quite on that level of understanding
Roothome is forbidden
I have shown hidden, and the partition is mounted with sudo, parted sees files in the directory 00, as it calculates 44gb over 80k files, but none visible to me
🙁 I'm quite comfortable in the bazzite innards from within, but the ways of the atomic magic (and how it works with grub) is over my head
That's quite appreciated.
As far as this level of issues goes, I think checkyourfax and wolfyreload are the most knowledgable, but you shouldn't ping them
Oo good catch, I was about to ping the skies above and was looking in their direction
I've tried to be methodical in this hoping I could pay it forwards to someone less knowledgeable who needs help on this in future. I know some are not as fortunate as I am to have means to backup use spare media before repaving. Thankb ou both, I'm going to take an image so perhaps I can revisit this in the event of further support or progress if other users get stuck.
:/ The weird thing is that nothing you described should lead to this state.
Yup, I think the other user linked in OP seemed to have not obviously messed anything that would cause me to assume they broke it.
Can you try going into grub (mash arrow keys while booting)
and add this to kernel parameters:
ostree.prepare-root.composefs=0
and pray it boots
you will need to put it somewhere on the line with quiet and splashthat's this menu?
Apologies, I am not familiar with editing grub.
you go press "e" on an entry
then you can edit the values there
you need to add it to the line with "linux"
and then add
ostree.prepare-root.composefs=0 after splash
it has to be on the same line, but grub will sometimes break up lines in two, meaning you add it to the apparent second line. Just add it after splashApologies, I don't see splash, only quiet. I do see the karg I added a few boots before this went haywire
after modprobe.blacklist=hic_uclogic
put it there
with a space in between
after you added it press F10
Should the karg I added be in there twice? I'm not familiar but it strikes me as odd the command is listed twice
its a little odd but its ok
doesn't matter
this means that it was added without an --append-if-missing flag in rpm-ostree
After f10 I am back in emergency mode, checking for mistakes..
Assuming I'm not supposed to head into emergency
reboot and try adding it again
ostree.prepare-root.composefs=0
Yeah its not there this time. F10 saves the changes?
make sure you don't accidentally add it on the initrd line
no nothing will be saved
this is temporary
i was just hoping this might succeed your boot
and you can try and mend the issue by rolling back
typo
composefs
not compsfs
make sure there are 0 typos
or it just wont work
ostree.prepare-root.composefs=0
Facepalm, apologies. Long working day and your patience has been appreciated.
this will disable the overlayfs that might be the cause of your failed boot
let me know if this works
if not i think your ostree got corrupted somehow
from a hardreset or something
Kicked into emergency mode after f10
what does it say this time?
Tried both bazzite 42 boot options same thing.
otherwise you can still try the other GRUB entry maybe? I'm sure you already tried that?
Tried both the bazzite 42 entries with the advised revision to grub
you can probably scrape your /home from your btrfs volume and reinstall, but I think this install is corrupted
you could try to check the ext4 and btrfs partition with gparted live
The only thing I can think that comes to mind as different, about a month ago I took a clonezilla image of my ssd to another ssd which was connected for a week or so, but has been disconnected for several weeks and boots since then, not sure if having it connected for a small time some time ago confused the boot order? I had something like this happen with windows over a decade ago, where the part of the boot ended up on another drive. But the timeline doesn't line up for that to make sense.
Can I bring the boot from an old clone and use it somehow on this image?
not in a supported way sadly
I found home via gparted so can try to copy out that way.
wait, could you try to take the extranous kernel arg?
out
and then try to boot?
It was an extravagant suggestion and I'm out of my depth, so appreciate this response
the one for uclogic
maybe its just confusing grub and screwing up all the other kernel args
that would fail boot
this is just a hunch
but maybe it will do something
I figured because the second boot option in grub didn't have the karg in the grub, that this was effectively doing that. But I can edit the first option too
i think its a red herring and its just that rpm-ostree corrupted your ostree
2nd image here
but ehh it won't hurt anything to try
I ran these kargs on my system to try and test something for another user I am supporting, as described here. Unfortunately having to repave my OS after trying a fix for them on my own machine is not going to give them confidence 🙂 I know however that I have no certainty the karg caused any of this.
https://discord.com/channels/1072614816579063828/1397547788610441296/1397674972276850822
not the karg
It didn't hurt to try, but we struck out on that, also nope, with or without the ostree command you provided (when removing the full karg lines
but the karg causes rpm-ostree to redeploy
It did
i wonder does the command
ostree fsck work in emergency shell?Ostree: command not found
yeah no okay
i think this can't be mended
@Kyle Gospo This is the second person in a short time with a broken system after an rpm-ostree commit? should i report this upstream?
I applaud all of your efforts and have learned some new tricks
can't hurt
GitHub
Multiple broken systems after rpm-ostree deployment. · Issue #5450...
Describe the bug Hi. We have multiple users (Bazzite) since a short time that have a broken system after changing something in rpm-ostree. In this case a user who appended a karg to their system an...
It is interesting to note that there are also erroneous fedora 41 entries in grub, when they should've been long gone since april or may
old ones don't just go away
sadly
only way would be to re-deploy your entire grub config which i don't recommend
1 of them is BLS the other one is legacy
its not an issue
the legacy ones are no longer being generated because bootupd is now used to make sure a system supports BLS
https://discord.com/channels/1072614816579063828/1400504671554109451/1400504671554109451
There might be another one
Well my issue has gotten 0 response so far
@j0rge here was one user with the same issue
as Jay
cc @Robert
I have been able to temporarily fix it by following the instructions over here: https://github.com/ostreedev/ostree/issues/2283#issuecomment-3036579977
The only part I am not able to do is actually fixing the boot loader entries to be consistent with the deployments I have, but maybe someone over here can help with that part.
GitHub
OSTree boot links can get out of sync · Issue #2283 · ostreedev/o...
We've had one instance recently in RHCOS where a system failed to boot with: systemd[1]: Starting OSTree Prepare OS/... ostree-prepare-root[887]: ostree-prepare-root: Couldn't find specifie...
hmm how does that help
it seemingly can't even find root
if it were able to do that then not having overlays in the actual system would help with many things
No
Doesn't help
ok yeah i didn't phrase the question well
how does this affect the initrd environment?
other than telling the initrd to not mount the overlay on the actual target system
does the initrd itself use overkays in some way?