Build actions

Original message was deleted

M2•5/12/24, 2:19 PM

We've predominantly used the buildah action for all of our builds.

However, our runners run Ubuntu 22.04 and the podman version is quite old. We've run into numerous issues where things will build locally since we are on a newer version of podman but fail in CI due to the version difference.

We've transitioned some of our workflows to using docker instead and some of our workflows are pinned to fedora 39 to avoid the buildah bugs.

In a few months 24.04 runners will be available but this cycle of old podman is likely to repeat.

There was a homebrew action for installing a newer podman, but unfortunately that seems to not quite work.

Should we consider moving our build actions to docker since that is actually upgraded on our runners and will result in a more consistent build process?

When I moved the local builds of bluefin to docker, I had to make some changes to the Containerfile since Docker cannot resolve a COPY --from with shell expansion variables. Otherwise, it should be a smooth transition.

M2•5/12/24, 2:20 PM

@bsherman @Noel @j0rge @Kyle Gospo @Robert (p5)

M2•5/12/24, 2:23 PM

@EyeCantCU

bsherman•5/12/24, 2:23 PM

I like the idea of sticking with podman, but the problem is the combination of support for podman/ubuntu on GitHub actions. Without GitHub offering RHEL based builders which support more current podman, I agree it’s possible we’ll hit this kind of stuff in the future.

bsherman•5/12/24, 2:24 PM

Side benefit, the docker build actions work more easily in local “act” runs and with Gitea actions

Robert•5/12/24, 2:25 PM

You can also install newer versions of Podman using this script, which is how CentOS builds their bootc images
https://github.com/rsturla/fedora-bootc-base/blob/main/.github%2Fworkflows%2Fbuild.yml#L35-L44

GitHub

fedora-bootc-base/.github/workflows/build.yml at main · rsturla/fed...

Contribute to rsturla/fedora-bootc-base development by creating an account on GitHub.

bsherman•5/12/24, 2:25 PM

I had to move to docker for ucore kmods already. There was no other option (except being taught by p5)

j0rge•5/12/24, 2:25 PM

I never understood why we don't use docker like normal people

Jj0rge I never understood why we don't use docker like normal people 😍

bsherman•5/12/24, 2:25 PM

Sounds like you are a solid “whatever works, my dudes”

EyeCantCU•5/12/24, 2:25 PM

Torn on this. A long time ago I investigated using the latest version of Podman on runners however there wasn't a very clean way of doing it. My vote goes to using Docker to avoid this toil. We can also leverage buildx and all of the other awesome stuff it provides

EEyeCantCU Torn on this. A long time ago I investigated using the latest version of Podman ...

bsherman•5/12/24, 2:26 PM

I agree. Updating the builder each build is a tech debt maintenance burden

bsherman•5/12/24, 2:26 PM

Toil

j0rge•5/12/24, 2:26 PM

And that's what everyone else is using, no reason to deviate

EyeCantCU•5/12/24, 2:27 PM

Exactly

M2•5/12/24, 2:28 PM

Right now config is using a fedora 39 container to avoid the tar bug.

Ucore is on docker.

Somehow bluefin isn't being bitten by the tar bug even though we untar things but yesterday when working with a better install method for brew it was blowing up all over the place.

I think most of my annoyance is that it's Ubuntu runners so we're kinda building outside of the expected ecosystem

EyeCantCU•5/12/24, 2:29 PM

A Fedora container can be used during the build process but there's numerous complications in handling it that way

j0rge•5/12/24, 2:30 PM

I can think of zero advantages to use fedora containers in ci, and half a dozen disadvantages lol

Robert•5/12/24, 2:30 PM

My only worry is we will want to include BIB and other RedHat services in our workflows, which are build for Podman.
Podman has some features that upstream (and possibly us) may use in the future
Like the magic they are doing with the bootc builds

Robert•5/12/24, 2:33 PM

IMO if we maintain a simple action "setup-podman" which installs Podman using aptapt,

brew

brew

or manually, we no longer run into the issue with Ubuntu shipping an outdated version, and we can still benefit from being in the RedHat ecosystem as much as possible

EyeCantCU•5/12/24, 2:34 PM

In that case... Maybe we should look into writing an action that leverages OBS to pull in the latest version?

https://software.opensuse.org//download.html?project=devel%3Akubic%3Alibcontainers%3Aunstable&package=podman

EyeCantCU•5/12/24, 2:35 PM

1 year ago... Never mind

Robert•5/12/24, 2:35 PM

I feel RedHat MUST support GitHub Actions for what they are building. I see no other way.
It's one of the most used CI services, and it doesn't make sense for them to only support GitLab with their new AI stuff

M2•5/12/24, 2:36 PM

I would hate to do deb packaging.

RRobert I feel RedHat MUST support GitHub Actions for what they are building. I see no ...

bsherman•5/12/24, 2:36 PM

Since when has Red Hat only done things that makes sense?

EyeCantCU•5/12/24, 2:37 PM

I was about to say, best case scenario would be Fedora GitHub runners

Robert•5/12/24, 2:37 PM

Or a first-party setup-podmansetup-podman action

EEyeCantCU I was about to say, best case scenario would be Fedora GitHub runners

bsherman•5/12/24, 2:38 PM

I think that was kind of my first point except I said RHEL

EyeCantCU•5/12/24, 2:38 PM

Yeah

bsherman•5/12/24, 2:38 PM

Summary so far: we all agree on problem

bsherman•5/12/24, 2:39 PM

Two possible solutions are really:

switch to docker
or we need a setup-podman action

EyeCantCU•5/12/24, 2:40 PM

Problem being retrieving the latest version of Podman or building it is ugly

bsherman•5/12/24, 2:40 PM

(I think GitHub bringing a new distribution for action runners is much less likely)

EyeCantCU•5/12/24, 2:41 PM

Now... I could package Podman for Wolfi and we could use that

M2•5/12/24, 2:41 PM

So how would that work?

EyeCantCU•5/12/24, 2:42 PM

We already have buildah

xyny•5/12/24, 2:43 PM

in bluebuild CI we use docker buildx with github caching and i'd 1000% recommend that to ublue as well

squashing images is the only part that doesn't work

EyeCantCU•5/12/24, 2:43 PM

Either we'd need to build a Buildah/Podman Wolfi container and run everything through that or I'd need a compelling justification to add it to our official image suite. We'd likely need to write our own build action

xyny•5/12/24, 2:44 PM

podman in a container is pretty slow

M2•5/12/24, 2:44 PM

I'm currently in the docker build camp.

We're supposed to be lazy and I'm unsure maintaining another build action is preferable

M2•5/12/24, 2:45 PM

But my hesitation is if we cannot take advantage of BIB and zstd in future

EEyeCantCU Either we'd need to build a Buildah/Podman Wolfi container and run everything th...

bsherman•5/12/24, 2:46 PM

As much as I appreciate these efforts. Running tools in containers in actions adds more complexity and I think that’s not the desired direction for this issue

Robert•5/12/24, 2:46 PM

I might just ask in Matrix something like "Given the new Bootc containers which expects people to extend the base images with custom layers, the issues with maintaining an up-to-date Ubuntu Podman package and the incompatibilities with the older Podman version and Fedora 40 images, do you have any plans internally to support other ways of setting up Podman in GitHub Actions?"

xyny•5/12/24, 2:48 PM

i'd imagine a custom repo / ppa would be the other way