Ran into two issues we d like to offer
Ran into two issues, we'd like to offer mainline kernels and device specific kernels for the ally
BWe can talk whenever... but I have some thoughts...
AI think you just need a copr no?
BPre-conditions I assume...
whatever custom kernels are required are available in some yum repo already (out of scope for akmods)
whatever custom kernels are required are available in some yum repo already (out of scope for akmods)
Bi'm sure i have other assumptions but my brain... not sure
Bi don't know how many custom configs you are wanting, but...
i could imagine building custom
then, build existing kmods similar to we do now, but with the new kernel/base-image stuff as an input for the Containerfile to build against the proper kernel
... thats kind of the basic idea which I figure is already in mind
i could imagine building custom
fedora-ostree-desktops/base images with the various swapped out kernels... as a matrix (parallel jobs) where said "base" images all get proper image/kernel version metadata assignedthen, build existing kmods similar to we do now, but with the new kernel/base-image stuff as an input for the Containerfile to build against the proper kernel
... thats kind of the basic idea which I figure is already in mind
BThe problems I'm already starting to think about (and @akdev ) even yesterday/last night...
i think our process is inefficient at best. both for akmods and main repos
so i think we need some architectural/process improvements for the builds before tackling more akmods (and kernel flavors of them) seems reasonable
i think our process is inefficient at best. both for akmods and main repos
so i think we need some architectural/process improvements for the builds before tackling more akmods (and kernel flavors of them) seems reasonable
Bmaybe the problems of main and akmods are a bit different, but my head is in
maincontext is
each build in the (massive) matrix does:
1. calculate tags for inputs
2. build foo-main image
3. build foo-nvidia image
4. publish, sign, etc (if not PR)
this is true for EACH runner, so there's a lot of double effort, and... its because we don't publish the intermediate images unless we are NOT in a PR build... which makes it a bit tricky for
main*-main and *-nvidia images:each build in the (massive) matrix does:
1. calculate tags for inputs
2. build foo-main image
3. build foo-nvidia image
4. publish, sign, etc (if not PR)
this is true for EACH runner, so there's a lot of double effort, and... its because we don't publish the intermediate images unless we are NOT in a PR build... which makes it a bit tricky for
foo-nvidia to depend on foo-main in a distinct job/step of the build workflowBan idea for solution:
split the
given they'd need distinct Containerfiles
i wonder if we could in
1. calculate tags
2. build
3. export the image to a tarball (
4. use github cache to cache that path with unique key
5. publish, sign, etc (if not PR)
then... in
1. calculate tags
2.
3. build
4. publish, sign, etc (if not PR)
split the
push-ghcr job into 2...build-main and build-nvidiagiven they'd need distinct Containerfiles
i wonder if we could in
build-main steps:1. calculate tags
2. build
foo-main image3. export the image to a tarball (
podman export) and store in a defined path (details fuzzy here)4. use github cache to cache that path with unique key
5. publish, sign, etc (if not PR)
then... in
build-nvidia steps:1. calculate tags
2.
podman import the foo-main image from cached tarball3. build
foo-nvidia image4. publish, sign, etc (if not PR)
BIF that approach is viable, i think we could also use it in akmods for custom-kernel builds which would need to get used for akmods...
and there's probably some more stuff that could be done to streamline there as well.
and there's probably some more stuff that could be done to streamline there as well.
B</crazy idea>
Bmore crazy idea...
in akmods this approach could be helpful even today, without custom kernels, as we could stage a cached "prepared base" image for use by the 3 "common" "nvidia-470" and "nvidia-535" builds so they don't all have to do that work.
in akmods this approach could be helpful even today, without custom kernels, as we could stage a cached "prepared base" image for use by the 3 "common" "nvidia-470" and "nvidia-535" builds so they don't all have to do that work.
Ball this said, I really think attempting to implement that last use case first might be easiest.
akmods buils are much smaller than main now...
the relevant changes to workflow model can be tested in a PR branch build.
and if it works, it could be immediately extended for custom kernels with some matrix magic
akmods buils are much smaller than main now...
the relevant changes to workflow model can be tested in a PR branch build.
and if it works, it could be immediately extended for custom kernels with some matrix magic
Band... i'm going back to my day job... but i'll probably reply to a ping in here 
AI’m gonna need time to process this 
KSame here, but this is great
KThanks for sharing your thoughts on this
B(edited my list of steps above to include
podman export and podman import and to say tarball since that matchs up with man pages)AOkay I’m going to put my first thoughts here, but I haven’t analyzed Sherman’s thoughts too deeply.
1. Ideally we would have one container per kmd rpm package to avoid download overhead from the consumers
2. Custom kernels would be out of scope for main imo - it’s supposed to be close to fedora. I don’t really think we need to create base images for custom kernels necessarily, I think we can rpm-ostree override the kernel package no?
3. akmods repo could build a matrix of kmod + kernel versions - this would work as long as we support pulling rpms from custom repos which seems easy to do
4. From client perspective they would do stuff like:
1. Ideally we would have one container per kmd rpm package to avoid download overhead from the consumers
2. Custom kernels would be out of scope for main imo - it’s supposed to be close to fedora. I don’t really think we need to create base images for custom kernels necessarily, I think we can rpm-ostree override the kernel package no?
3. akmods repo could build a matrix of kmod + kernel versions - this would work as long as we support pulling rpms from custom repos which seems easy to do
4. From client perspective they would do stuff like:
BThis is 100% complementary to my idea.
If we had per-kmod images (eg ghcr.io/ublue-os/akmods/xpadneo) we could even tag them in a way that is easy to consume by consumer
eg,
all that said, i see splitting kmod builds into a matrix and splitting up the resulting images to be the easy bit which first requires the hard work of building cached "pre-prepared, ready-to-build a kmod base" images with distinct kernels
If we had per-kmod images (eg ghcr.io/ublue-os/akmods/xpadneo) we could even tag them in a way that is easy to consume by consumer
eg,
xpadneo:37, xpadneo:38, xpadneo:39 would be latest stock fedora kernels from the upstream images xpadneo:ally123 latest build for some kernel specific to allyxpadneo:customkern2 latest build for some other custom kernelall that said, i see splitting kmod builds into a matrix and splitting up the resulting images to be the easy bit which first requires the hard work of building cached "pre-prepared, ready-to-build a kmod base" images with distinct kernels
AI think the process for that is very simple no? Install the akmod stuff and then swap the kernel to the custom one
AOh I see
AI think the kernel base images should be in the repo where the custom kernel is built?
Bhmm... i'm trying to think how to explain this better
AI think I got your point
AWe need to build many kmods for any given kernel version
AAnd downloading the kernel each time is wasteful
AWe could prepackage these images once before the builds start
Band installing even build deps each time is wasteful (there is a
build-prep.sh in akmods repo which already does this per build today)
Byes, or pre-package a "ready-to-build-kmods-on-kernel-XYZ" image per kernel version at the start of each 'akmods' run
Bi think saying same thing
AYeah
KTwo things to figure out then
K1: how to make it as easy as possible for people to swap the kernel version, we need to find some way to either reinstall the proper kmods or easily uninstall all and reinstall what's needed
K2: how to make it easy for people to fork the akmods repo if they need to add additional kernels
KDownstream that is
Bi don't see this as being easy, in fact the more we add to akmods and main repos, the more complicated it becomes
Bi wrote "rebase" before, but i deleted to write a more complete response.
i do think swapping kernels for bazzite specifically can be a rebase which can be incorporated into yafti/installer/whatever-you-want
unistall/reinstall... other than rebase... uh... that's a different problem...
if we want users to be able to layer kmods which are built in akmods, that's a problem which we already have, so i think it's just a question of what to tackle first
i do think swapping kernels for bazzite specifically can be a rebase which can be incorporated into yafti/installer/whatever-you-want
unistall/reinstall... other than rebase... uh... that's a different problem...
if we want users to be able to layer kmods which are built in akmods, that's a problem which we already have, so i think it's just a question of what to tackle first
BUgh... the cache idea may not work
A repository can have up to 10GB of caches. Once the 10GB limit is reached, older caches will be evicted based on when the cache was last accessed.per: https://github.com/actions/cache#cache-limits
BI think instead of "cache" what I really want is "artifacts"
Bok, yep... artifacts work a treat, this is exactly what i wanted when i was talking about cache earlier
Band it's what they are meant for 
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts#passing-data-between-jobs-in-a-workflow
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts#passing-data-between-jobs-in-a-workflowBi did a test run where i build an image, save it to a tar file, upload, download, load it, and build FROM it
https://github.com/bsherman/cache-tests/actions/runs/6032455190/job/16367660838
https://github.com/bsherman/cache-tests/actions/runs/6032455190/job/16367660838
BGitHub
The goal of this PR is to streamline image builds.
The idea is to split *-main and *-nvidia image builds into 2 jobs.
The *-main job will run first, and each matrix job instance will build the foo-...
The idea is to split *-main and *-nvidia image builds into 2 jobs.
The *-main job will run first, and each matrix job instance will build the foo-...
Bugh, the artifacts is feeling pretty painful, these tarballs are so big
BGitHub
The goal of this PR is to streamline image builds.
Current problem:
both *-main and *-nvidia images are built once each for every permutation of a matrix comprised of: fedora major version (37, 38)...
Current problem:
both *-main and *-nvidia images are built once each for every permutation of a matrix comprised of: fedora major version (37, 38)...
BI think we may just need to use the registry for this. and then my artifacts PR https://github.com/ublue-os/main/pull/325 (but using registry push instead of artifacts) and then we'd have much cleaner builds here.
but... there's the whole security concern... how can we mitigate the security concern?
- push, but don't sign, a PR build?
- use a distinct-from-normal image name (that would get challenging, I think)
- use PR tags only (this already happens, except in the case of bugs in workflow)
but... there's the whole security concern... how can we mitigate the security concern?
- push, but don't sign, a PR build?
- use a distinct-from-normal image name (that would get challenging, I think)
- use PR tags only (this already happens, except in the case of bugs in workflow)
AUsing artifacts will probably be very slow
Bthis may sound unrelated to the original akmods question, but it's not because it's trying to solve the basic problem of large-ish matrix builds with multi-gig shared artifacts (container images)
Bthat's what i found above yes
yes