GAre you by any chance from Bulgaria?
IWe just found an installation corruption error. Information: We base our images on ublue-os images. We deploy hardened_malloc. One variant (cosmic) has a permanent corruption issue. ostree is unable to do any image layering operations. The ostree filesystem also encounters errors (ostree fsck). We currently believe that the error is at ostree or rechunks. Our hardened_malloc seems to find this error but it is possible that more images are affected by this. I am unable to pinpoint the error. "ostree: fatal allocator error: detected write after free"
IGitHub![[BUG] rpm-ostree can't deploy on Cosmic images · Issue #369 · secur...](https://images-ext-1.discordapp.net/external/xSqebdFEXpPtRo275PkFd-byRq8uuRckPWOTDGNJ85s/https/opengraph.githubassets.com/e5bfaab463f0e91530c066026854191ddccfc7c2532099b1345bdb3d4d26572d/secureblue/secureblue/issues/369)
Describe the bug On the cosmic-main-hardened and cosmic-main-userns-hardened images (and probably nvidia images too), rpm-ostree can't write new deployments. Attempting to install or uninstall ...
II would like to help find the cause. But I do not know what combination is causing it. It looks related to rechunks + ostree.
AHi, slowly but surely we will fix jt
ACan you also find the line in ostree that emits this error
IIf you tell me how, yes. I am unsure what you mean
AI guess there is no line
AThis command is emitted by malloc
ASorry the error
IYes. Our malloc implementation does not like memory corruption bugs.
AWhat does filesystem corruption mean
IThe command: "sudo ostree fsck" finds issues in layers. "sudo ostree --delete" is unable to repair them. It may have nothing to do with the issue. Maybe just circumstance.
AAug 12 15:49:46 localhost rpm-ostree[13779]: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
ASeems like the error is this
AThis is emitted by rpm ostree so we should be able to find the line
ACan you point to the file?
AThis causes the core dump
IPlease explain. I am unsure what exactly I should do. (Just FYI I need to reinstall to be able to debug this further, tried to repair it and may have caused more issues in ostree. Just to be able to pinpoint it I would need to reinstall to make sure that I do not find other issues that I caused myself.)
IThe github issue report is from an other user. I can confirm I had the same issue.
ACan you get some extra errors from ostree fsck?
AProbably rpm-ostree dying caused this
IYes, that is what I also believe. My working theory is that ostree + rechunks have one bug. Our malloc finds this issue. Nobody else sees it because everyone else uses the glic malloc instead of hardened_malloc. It is possible that we are affected because our malloc does not like memory corruption. It causes rpm ostree to crash. But that may also mean that ostree or rechunks have an issue that nobody else sees.
II will try to debug this again in a few hours and report back. I am busy right now.
Aas part of rechunk i replace some rpm-ostree files to make space
Athey never seemed to have made an issue with layering
Abut perhaps they do
Ahttps://github.com/hhd-dev/rechunk/blob/e46fb40a61366795e65a1ac7411799515294c98a/1_prune.sh#L157 it is this line over here
IIf you want to test this for yourself you may deploy our "cosmic-main-userns-hardened" image from https://github.com/secureblue/secureblue?tab=readme-ov-file#installation - I can give you logs later.
GitHub
Fedora Atomic images for GNOME, KDE Plasma, Bluefin, Sway, Cinnamon, Wayfire, River, and Hyprland with some hardening applied - secureblue/secureblue
IThe important part seems to be the hardened_malloc + rechunks. This is a bit over my head. I only see the issue but do not know how to fix it.
IAlso do we want to move this discussion somewhere else if it is more pinpointed? I would open an issue report but I do not know where yet.
Atime to brick my vm
II do not believe that hardened_malloc is the issue. Because our non rechunked images work fine. And hardened_malloc just enforces certain malloc related rules.
FYou can make a Discord thread to make discussion in 1 place
Afor some reason i apepar to have ran out of ram so i cant start my vm
Aand im too bothered to reboot it seems
IIS this only possible for new messages or can I mark and move the already send messages to a thread?
A7 day vivaldi memory leak goes brrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
FI'm not sure if messages can be moved
IYou can just post further messages to our issue report here: https://github.com/secureblue/secureblue/issues/369 - It is the simplest solution until we know more. Thank you for taking time to debug this
GitHub
Describe the bug On the cosmic-main-hardened and cosmic-main-userns-hardened images (and probably nvidia images too), rpm-ostree can't write new deployments. Attempting to install or uninstall ...
Agive me a rebase command
Aam confused
Aim rebasing to his image
Iyou want to follow the instructions here: https://github.com/secureblue/secureblue?tab=readme-ov-file#rebasing-recommended - and append "cosmic-main-userns-hardened" in the command
GitHub
Fedora Atomic images for GNOME, KDE Plasma, Bluefin, Sway, Cinnamon, Wayfire, River, and Hyprland with some hardening applied - secureblue/secureblue
Ai will point out that we might have slight CAP issues with rechunk
Aso you might need to do some AB testing of the images
Ai can give you a command that will spit the differences between t2 images
ISounds good.
IYou may need to wait until a new image is build to see the issue for yourself. If you rebase now you can not upgrade to a newer version. And the issue is only caused in rpm-ostree operations with hardened_malloc. You may not see any issue before the conditions are met. Conditions: rechunks + hardened_malloc + sys upgrade available
II can ask our main dev to force a rebuild after you have deployed the image to speed up this process.
