Fedora 43 Kernel Build Issues

Think we must be doing something wrong in akmods F43. Can't reproduce the error reinstalling the kernel from Koji
35 Replies
Robert
RobertOP6d ago
FROM quay.io/fedora-ostree-desktops/silverblue:43


ARG KERNEL_VERSION=6.17.1-300.fc43.x86_64

RUN dnf remove -y kernel{,-core,-modules,-modules-core,-modules-extra,-tools,-tools-libs}
RUN rm -rf /usr/lib/modules/*
RUN KERNEL_MAJOR_MINOR_PATCH=$(echo $KERNEL_VERSION | cut -d '-' -f 1) && \
KERNEL_RELEASE=$(echo $KERNEL_VERSION | cut -d '-' -f 2 | rev | cut -d '.' -f 2- | rev) && \
ARCH=$(echo $KERNEL_VERSION | rev | cut -d '.' -f 1 | rev) && \
dnf install -y \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-core-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-modules-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-modules-core-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-modules-extra-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm
RUN dnf versionlock add $(rpm -qa --queryformat '%{NAME}-%{VERSION}-%{RELEASE}\n' kernel*)
FROM quay.io/fedora-ostree-desktops/silverblue:43


ARG KERNEL_VERSION=6.17.1-300.fc43.x86_64

RUN dnf remove -y kernel{,-core,-modules,-modules-core,-modules-extra,-tools,-tools-libs}
RUN rm -rf /usr/lib/modules/*
RUN KERNEL_MAJOR_MINOR_PATCH=$(echo $KERNEL_VERSION | cut -d '-' -f 1) && \
KERNEL_RELEASE=$(echo $KERNEL_VERSION | cut -d '-' -f 2 | rev | cut -d '.' -f 2- | rev) && \
ARCH=$(echo $KERNEL_VERSION | rev | cut -d '.' -f 1 | rev) && \
dnf install -y \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-core-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-modules-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-modules-core-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm \
https://kojipkgs.fedoraproject.org//packages/kernel/$KERNEL_MAJOR_MINOR_PATCH/$KERNEL_RELEASE/$ARCH/kernel-modules-extra-$KERNEL_MAJOR_MINOR_PATCH-$KERNEL_RELEASE.$ARCH.rpm
RUN dnf versionlock add $(rpm -qa --queryformat '%{NAME}-%{VERSION}-%{RELEASE}\n' kernel*)
The above Containerfile works Is anyone else able to reproduce the install failure? The workflow spans two repos and it's not trivial to track down how it works
antheas
antheas6d ago
would help to have the full error message github actions cuts it well dnf does
Robert
RobertOP5d ago
I think it's "Invalid cross-device link" @bsherman Installing the kernel from the RPMs bundled in the akmods OCI causes builds to fail in postinstall scripts.
Installing the kernel directly from Koji (i.e. the above) works.
I don't understand the magic in akmods enough to know why this may be happening Failed build log: https://github.com/ublue-os/main/actions/runs/18515584599/job/52765149589?pr=1155#step:5:510 It happens to all F43 images, and no other F42/F41 images have the issue
bsherman
bsherman5d ago
ok, so the failure happens in a bluefin/aurora using the akmods provided kernel/kmods? oh, i see
Robert
RobertOP5d ago
In ublue-os/main currently, though I would assume any that use kernel-cache We've not got to the bluefin/aurora part yet 🙂
bsherman
bsherman5d ago
hmm... odd. i'd forgotten we used kernel-cache kernels in main so the problem is when it runs dracut i'm setting up to reproduce this
Robert
RobertOP5d ago
What are you doing differently than the above? Are you 'rpmrebuild'ing it? Whatever that does...
bsherman
bsherman5d ago
i'm just wanting to run it local, because 1) it's faster to iterate, even for just adding extra debug output 2) sometimes it seems different output is available local, though unlikely in this case the way main/akmods work now is you run the devcontainer, then you can run exactly the same commands as run in the CI so that's solid what is kernel-0 package? https://github.com/ublue-os/main/actions/runs/18515584599/job/52765149589?pr=1155#step:5:505
antheas
antheas5d ago
kernel-0...:versionsomething or not hm
bsherman
bsherman5d ago
yeah, that's new... not in F42 oh, it's just the kernel package's internal name hmm... i wonder if would work to bypass the kernel-install as run by the rpm install scriptlet? that at least let me discover we want packages not in F43 No match for argument: mesa-libxatracker No match for argument: pipewire-libs-extra
Robert
RobertOP5d ago
Both of those IMO are non-critical and can be disabled for F43
bsherman
bsherman5d ago
agreed, i've made the packages.json \change for local testing do you have a link handy for the fedora kernel package source? i'd like to read the scriptlet for kernel install blah
>>> Running %triggerin scriptlet: systemd-0:258-1.fc43.x86_64
>>> Finished %triggerin scriptlet: systemd-0:258-1.fc43.x86_64
>>> Scriptlet output:
>>> Failed to connect to audit log, ignoring: Invalid argument
>>>
Transaction failed: Rpm transaction failed.
>>> Running %triggerin scriptlet: systemd-0:258-1.fc43.x86_64
>>> Finished %triggerin scriptlet: systemd-0:258-1.fc43.x86_64
>>> Scriptlet output:
>>> Failed to connect to audit log, ignoring: Invalid argument
>>>
Transaction failed: Rpm transaction failed.
Robert
RobertOP5d ago
$ dnf download --source kernel
$ rpm2cpio <rpm>
$ <untar and open tar.gz
$ dnf download --source kernel
$ rpm2cpio <rpm>
$ <untar and open tar.gz
There's probably ways to do that in 1 command, but meh
Robert
RobertOP5d ago
Didn't link to the Bazzite one as I didn't know how much you changed it
antheas
antheas5d ago
did not touch the post scripts although wyd the post script that blows up is rpm-ostree?
Robert
RobertOP5d ago
There's a /usr/lib/kernel/install.d script on all systems which gives custom scripts needed during kernel installs That calls rpm-ostree kernel-install
bsherman
bsherman5d ago
right, of course the new problem is later...
antheas
antheas5d ago
whats the problem now?
bsherman
bsherman5d ago
Running %triggerin scriptlet: systemd-0:258-1.fc43.x86_64 Finished %triggerin scriptlet: systemd-0:258-1.fc43.x86_64 Scriptlet output: Failed to connect to audit log, ignoring: Invalid argument Transaction failed: Rpm transaction failed.
possibly something else, but this is the last output maybe we shim /usr/lib/kernel/install.d/{05-rpmostree.install,50-dracut.install} so they do nothing... as we shouldn't need either of those to function at kernel install time in container image, because we later run dracut ourselves i'll try that, then commit to branch, see you see anything with the other issues, because I'm not seeing it @Robert i pushed the merge from my local and oh 🙂 you've been fixing it too
Robert
RobertOP5d ago
I'm trying... But the webui isn't the best...
If you have it sorted, feel free to force push 🙂
bsherman
bsherman5d ago
except, invalid json yeah, web UI is a pain 🙁 k, i'll force push ok, so at this point, I don't expect the build to succeed, but it does get further, and i have comments to give context for how/why i solved the kernel/dracut issue the way i did but i have yet to understand the exact failure on packages.sh installs, which is where i expect it to fail now i'm going to have to log off for some family stuff, but it's progress at least
Robert
RobertOP5d ago
Thank you for the help!
bsherman
bsherman5d ago
you're welcome! we'll see if i can offer more tonight or not... probably i can offer some tomorrow even if not tonight yep, this is what i saw locally: https://github.com/ublue-os/main/actions/runs/18544913559/job/52860828561#step:5:1313
Robert
RobertOP5d ago
Is the main source of the error android-udev-rules? I can't see anything else trying to use systemd in that portion of the logs
bsherman
bsherman5d ago
maybe... perhaps we can shim out systemd? it's pretty pointless in container is android-udev-rules our package? doesn't look like it
Robert
RobertOP5d ago
It's installed from our staging COPR
android-udev-rules noarch 20241019-1.fc42 copr:copr.fedorainfracloud.org:ublue-os:staging 73.6 KiB
android-udev-rules noarch 20241019-1.fc42 copr:copr.fedorainfracloud.org:ublue-os:staging 73.6 KiB
But I can't see it in ublue-os/packages Ah https://github.com/ublue-os/android-udev-rules Though no spec files
bsherman
bsherman5d ago
oh, it's a fork, and it's built in copr, i think well i think we could probably just shim things more i'm curious if we can shim kernel-install and systemd and that would cleanup a lot of our warnings the only thing kernel-install may do which i didn't want to skip is add/remove of files in /usr/lib/modules when upgrading i'll test a systemd shim quick
Robert
RobertOP5d ago
I'm removing the android udev rules package and seeing if it helps
bsherman
bsherman5d ago
i'll do anything i'm doing locally for now, so no worries
Robert
RobertOP5d ago
So will I... Just 10x slower than you 😆
No description
bsherman
bsherman5d ago
ok, i'll look at that later, gotta go
Robert
RobertOP5d ago
So removing the package worked locally and we have a completed build. Will commit that for now just to get it building. It may be a critical package for Bazzite, but it also probably doesn't need to be an RPM / have postinstall scripts.
bsherman
bsherman5d ago
I can't remember, but part of me thinks that package was more for android development than anything else... but yeah, that should be explored separately
antheas
antheas4d ago
has some udev rules so adb works if you can pull the rules directly, its probably better lets start nixing random rpms

Did you find this page helpful?