Universal Blue•3y ago•

188 replies

Ran into two issues we d like to offer

Ran into two issues, we'd like to offer mainline kernels and device specific kernels for the ally

bsherman•8/30/23, 2:46 PM

We can talk whenever... but I have some thoughts...

akdev•8/30/23, 2:47 PM

I think you just need a copr no?

bsherman•8/30/23, 2:48 PM

Pre-conditions I assume...

whatever custom kernels are required are available in some yum repo already (out of scope for akmods)

bsherman•8/30/23, 2:48 PM

i'm sure i have other assumptions but my brain... not sure

bsherman•8/30/23, 2:52 PM

i don't know how many custom configs you are wanting, but...

i could imagine building custom fedora-ostree-desktops/basefedora-ostree-desktops/base images with the various swapped out kernels... as a matrix (parallel jobs) where said "base" images all get proper image/kernel version metadata assigned

then, build existing kmods similar to we do now, but with the new kernel/base-image stuff as an input for the Containerfile to build against the proper kernel

... thats kind of the basic idea which I figure is already in mind

bsherman•8/30/23, 2:53 PM

The problems I'm already starting to think about (and @akdev ) even yesterday/last night...

i think our process is inefficient at best. both for akmods and main repos

so i think we need some architectural/process improvements for the builds before tackling more akmods (and kernel flavors of them) seems reasonable

bsherman•8/30/23, 2:55 PM

maybe the problems of main and akmods are a bit different, but my head is in

main

main

at the moment, so here's my thought there, and i think it could be relevant to how we could solve multiple kernel-images here

bsherman•8/30/23, 2:58 PM

context is

main

main

repo, which now builds both *-main*-main and *-nvidia*-nvidia images:

each build in the (massive) matrix does:
1. calculate tags for inputs
2. build foo-main image
3. build foo-nvidia image
4. publish, sign, etc (if not PR)

this is true for EACH runner, so there's a lot of double effort, and... its because we don't publish the intermediate images unless we are NOT in a PR build... which makes it a bit tricky for foo-nvidiafoo-nvidia to depend on foo-mainfoo-main in a distinct job/step of the build workflow

bsherman•8/30/23, 3:02 PM

an idea for solution:
split the push-ghcrpush-ghcr job into 2...
build-mainbuild-main and build-nvidiabuild-nvidia
given they'd need distinct Containerfiles

i wonder if we could in build-mainbuild-main steps:
1. calculate tags
2. build foo-mainfoo-main image
3. export the image to a tarball (podman exportpodman export) and store in a defined path (details fuzzy here)
4. use github cache to cache that path with unique key
5. publish, sign, etc (if not PR)

then... in build-nvidiabuild-nvidia steps:
1. calculate tags
2. podman importpodman import the foo-mainfoo-main image from cached tarball
3. build foo-nvidiafoo-nvidia image
4. publish, sign, etc (if not PR)

bsherman•8/30/23, 3:04 PM

IF that approach is viable, i think we could also use it in akmods for custom-kernel builds which would need to get used for akmods...
and there's probably some more stuff that could be done to streamline there as well.

bsherman•8/30/23, 3:04 PM

</crazy idea>

bsherman•8/30/23, 3:06 PM

more crazy idea...

in akmods this approach could be helpful even today, without custom kernels, as we could stage a cached "prepared base" image for use by the 3 "common" "nvidia-470" and "nvidia-535" builds so they don't all have to do that work.

bsherman•8/30/23, 3:08 PM

all this said, I really think attempting to implement that last use case first might be easiest.

akmods buils are much smaller than main now...

the relevant changes to workflow model can be tested in a PR branch build.

and if it works, it could be immediately extended for custom kernels with some matrix magic

bsherman•8/30/23, 3:09 PM

and... i'm going back to my day job... but i'll probably reply to a ping in here

akdev•8/30/23, 3:11 PM

I’m gonna need time to process this

Kyle GospoOP•8/30/23, 3:17 PM

Same here, but this is great

Kyle GospoOP•8/30/23, 3:17 PM

Thanks for sharing your thoughts on this

bsherman•8/30/23, 3:24 PM

(edited my list of steps above to include podman exportpodman export and podman importpodman import and to say tarballtarball since that matchs up with man pages)

akdev•8/30/23, 3:25 PM

Okay I’m going to put my first thoughts here, but I haven’t analyzed Sherman’s thoughts too deeply.

1. Ideally we would have one container per kmd rpm package to avoid download overhead from the consumers
2. Custom kernels would be out of scope for main imo - it’s supposed to be close to fedora. I don’t really think we need to create base images for custom kernels necessarily, I think we can rpm-ostree override the kernel package no?
3. akmods repo could build a matrix of kmod + kernel versions - this would work as long as we support pulling rpms from custom repos which seems easy to do
4. From client perspective they would do stuff like:

rpm-ostree override remove kernel —install kernel-custom 
COPY —from=ghcr.io/ublue-os/akmods/xpadneo:custom /tmp/akmods /tmp

rpm-ostree override remove kernel —install kernel-custom 
COPY —from=ghcr.io/ublue-os/akmods/xpadneo:custom /tmp/akmods /tmp

bsherman•8/30/23, 3:34 PM

This is 100% complementary to my idea.

If we had per-kmod images (eg ghcr.io/ublue-os/akmods/xpadneo) we could even tag them in a way that is easy to consume by consumer

eg, xpadneo:37, xpadneo:38, xpadneo:39xpadneo:37, xpadneo:38, xpadneo:39 would be latest stock fedora kernels from the upstream images
xpadneo:ally123xpadneo:ally123 latest build for some kernel specific to ally
xpadneo:customkern2xpadneo:customkern2 latest build for some other custom kernel

all that said, i see splitting kmod builds into a matrix and splitting up the resulting images to be the easy bit which first requires the hard work of building cached "pre-prepared, ready-to-build a kmod base" images with distinct kernels

akdev•8/30/23, 3:40 PM

I think the process for that is very simple no? Install the akmod stuff and then swap the kernel to the custom one

akdev•8/30/23, 3:41 PM

Oh I see

akdev•8/30/23, 3:42 PM

I think the kernel base images should be in the repo where the custom kernel is built?

bsherman•8/30/23, 3:42 PM

hmm... i'm trying to think how to explain this better

akdev•8/30/23, 3:42 PM

I think I got your point

akdev•8/30/23, 3:42 PM

We need to build many kmods for any given kernel version

akdev•8/30/23, 3:43 PM

And downloading the kernel each time is wasteful

akdev•8/30/23, 3:43 PM

We could prepackage these images once before the builds start

bsherman•8/30/23, 3:43 PM

and installing even build deps each time is wasteful (there is a build-prep.shbuild-prep.sh in akmodsakmods repo which already does this per build today)

Aakdev We could prepackage these images once before the builds start

bsherman•8/30/23, 3:45 PM

yes, or pre-package a "ready-to-build-kmods-on-kernel-XYZ" image per kernel version at the start of each 'akmods' run

bsherman•8/30/23, 3:45 PM

i think saying same thing

akdev•8/30/23, 3:45 PM

Yeah

Kyle GospoOP•8/30/23, 5:32 PM

Two things to figure out then

Kyle GospoOP•8/30/23, 5:32 PM

1: how to make it as easy as possible for people to swap the kernel version, we need to find some way to either reinstall the proper kmods or easily uninstall all and reinstall what's needed

Kyle GospoOP•8/30/23, 5:32 PM

2: how to make it easy for people to fork the akmods repo if they need to add additional kernels

Kyle GospoOP•8/30/23, 5:33 PM

Downstream that is

KKyle Gospo 2: how to make it easy for people to fork the akmods repo if they need to add ad...

bsherman•8/30/23, 5:53 PM

i don't see this as being easy, in fact the more we add to akmods and main repos, the more complicated it becomes

KKyle Gospo 1: how to make it as easy as possible for people to swap the kernel version, we ...

bsherman•8/30/23, 5:55 PM

i wrote "rebase" before, but i deleted to write a more complete response.

i do think swapping kernels for bazzite specifically can be a rebase which can be incorporated into yafti/installer/whatever-you-want

unistall/reinstall... other than rebase... uh... that's a different problem...

if we want users to be able to layer kmods which are built in akmods, that's a problem which we already have, so i think it's just a question of what to tackle first

bsherman•8/31/23, 1:51 AM

Ugh... the cache idea may not work

A repository can have up to 10GB of caches. Once the 10GB limit is reached, older caches will be evicted based on when the cache was last accessed.

per: https://github.com/actions/cache#cache-limits

bsherman•8/31/23, 2:11 AM

I think instead of "cache" what I really want is "artifacts"

bsherman•8/31/23, 2:47 AM

ok, yep... artifacts work a treat, this is exactly what i wanted when i was talking about cache earlier

bsherman•8/31/23, 2:48 AM

and it's what they are meant for

https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts#passing-data-between-jobs-in-a-workflow

bsherman•8/31/23, 2:49 AM

i did a test run where i build an image, save it to a tar file, upload, download, load it, and build FROM it
https://github.com/bsherman/cache-tests/actions/runs/6032455190/job/16367660838

bsherman•8/31/23, 3:31 AM

this is NOT ready yet, but i'm working on it: https://github.com/ublue-os/main/pull/325

GitHub

feat: use artifacts to streamline image builds by bsherman · Pull R...

The goal of this PR is to streamline image builds.
The idea is to split *-main and *-nvidia image builds into 2 jobs.
The *-main job will run first, and each matrix job instance will build the foo-...

bsherman•8/31/23, 4:10 AM

ugh, the artifacts is feeling pretty painful, these tarballs are so big

bsherman•8/31/23, 1:21 PM

This is a different attempt ... still not happy

https://github.com/ublue-os/main/pull/326

GitHub

feat: split main/nvidia into distinct steps within job by bsherman ...

The goal of this PR is to streamline image builds.
Current problem:
both *-main and *-nvidia images are built once each for every permutation of a matrix comprised of: fedora major version (37, 38)...

bsherman•8/31/23, 1:24 PM

I think we may just need to use the registry for this. and then my artifacts PR https://github.com/ublue-os/main/pull/325 (but using registry push instead of artifacts) and then we'd have much cleaner builds here.

but... there's the whole security concern... how can we mitigate the security concern?

- push, but don't sign, a PR build?
- use a distinct-from-normal image name (that would get challenging, I think)
- use PR tags only (this already happens, except in the case of bugs in workflow)

akdev•8/31/23, 1:26 PM

Using artifacts will probably be very slow

bsherman•8/31/23, 1:26 PM

this may sound unrelated to the original akmods question, but it's not because it's trying to solve the basic problem of large-ish matrix builds with multi-gig shared artifacts (container images)

Aakdev Using artifacts will probably be very slow

bsherman•8/31/23, 1:26 PM

that's what i found above

yes

Ran into two issues we d like to offer

Similar Threads

Similar Threads

Similar Threads