MWe've predominantly used the buildah action for all of our builds.
However, our runners run Ubuntu 22.04 and the podman version is quite old. We've run into numerous issues where things will build locally since we are on a newer version of podman but fail in CI due to the version difference.
We've transitioned some of our workflows to using docker instead and some of our workflows are pinned to fedora 39 to avoid the buildah bugs.
In a few months 24.04 runners will be available but this cycle of old podman is likely to repeat.
There was a homebrew action for installing a newer podman, but unfortunately that seems to not quite work.
Should we consider moving our build actions to docker since that is actually upgraded on our runners and will result in a more consistent build process?
When I moved the local builds of bluefin to docker, I had to make some changes to the Containerfile since Docker cannot resolve a COPY --from with shell expansion variables. Otherwise, it should be a smooth transition.
However, our runners run Ubuntu 22.04 and the podman version is quite old. We've run into numerous issues where things will build locally since we are on a newer version of podman but fail in CI due to the version difference.
We've transitioned some of our workflows to using docker instead and some of our workflows are pinned to fedora 39 to avoid the buildah bugs.
In a few months 24.04 runners will be available but this cycle of old podman is likely to repeat.
There was a homebrew action for installing a newer podman, but unfortunately that seems to not quite work.
Should we consider moving our build actions to docker since that is actually upgraded on our runners and will result in a more consistent build process?
When I moved the local builds of bluefin to docker, I had to make some changes to the Containerfile since Docker cannot resolve a COPY --from with shell expansion variables. Otherwise, it should be a smooth transition.
M@bsherman @Noel @j0rge @Kyle Gospo @Robert (p5)
M@EyeCantCU
BI like the idea of sticking with podman, but the problem is the combination of support for podman/ubuntu on GitHub actions. Without GitHub offering RHEL based builders which support more current podman, I agree it’s possible we’ll hit this kind of stuff in the future.
BSide benefit, the docker build actions work more easily in local “act” runs and with Gitea actions
RYou can also install newer versions of Podman using this script, which is how CentOS builds their bootc images
https://github.com/rsturla/fedora-bootc-base/blob/main/.github%2Fworkflows%2Fbuild.yml#L35-L44
https://github.com/rsturla/fedora-bootc-base/blob/main/.github%2Fworkflows%2Fbuild.yml#L35-L44
GitHub
Contribute to rsturla/fedora-bootc-base development by creating an account on GitHub.
BI had to move to docker for ucore kmods already. There was no other option (except being taught by p5)
JI never understood why we don't use docker like normal people 
BSounds like you are a solid “whatever works, my dudes”
ETorn on this. A long time ago I investigated using the latest version of Podman on runners however there wasn't a very clean way of doing it. My vote goes to using Docker to avoid this toil. We can also leverage buildx and all of the other awesome stuff it provides
BI agree. Updating the builder each build is a tech debt maintenance burden
BToil
JAnd that's what everyone else is using, no reason to deviate
EExactly
MRight now config is using a fedora 39 container to avoid the tar bug.
Ucore is on docker.
Somehow bluefin isn't being bitten by the tar bug even though we untar things but yesterday when working with a better install method for brew it was blowing up all over the place.
I think most of my annoyance is that it's Ubuntu runners so we're kinda building outside of the expected ecosystem
Ucore is on docker.
Somehow bluefin isn't being bitten by the tar bug even though we untar things but yesterday when working with a better install method for brew it was blowing up all over the place.
I think most of my annoyance is that it's Ubuntu runners so we're kinda building outside of the expected ecosystem
EA Fedora container can be used during the build process but there's numerous complications in handling it that way
JI can think of zero advantages to use fedora containers in ci, and half a dozen disadvantages lol
RMy only worry is we will want to include BIB and other RedHat services in our workflows, which are build for Podman.
Podman has some features that upstream (and possibly us) may use in the future
Like the magic they are doing with the bootc builds
Podman has some features that upstream (and possibly us) may use in the future
Like the magic they are doing with the bootc builds
RIMO if we maintain a simple action "setup-podman" which installs Podman using
apt, brew or manually, we no longer run into the issue with Ubuntu shipping an outdated version, and we can still benefit from being in the RedHat ecosystem as much as possibleEIn that case... Maybe we should look into writing an action that leverages OBS to pull in the latest version?
https://software.opensuse.org//download.html?project=devel%3Akubic%3Alibcontainers%3Aunstable&package=podman
https://software.opensuse.org//download.html?project=devel%3Akubic%3Alibcontainers%3Aunstable&package=podman
E1 year ago... Never mind
RI feel RedHat MUST support GitHub Actions for what they are building. I see no other way.
It's one of the most used CI services, and it doesn't make sense for them to only support GitLab with their new AI stuff
It's one of the most used CI services, and it doesn't make sense for them to only support GitLab with their new AI stuff
MI would hate to do deb packaging.
BSince when has Red Hat only done things that makes sense?
EI was about to say, best case scenario would be Fedora GitHub runners
ROr a first-party
setup-podman actionBI think that was kind of my first point except I said RHEL
EYeah
BSummary so far: we all agree on problem
BTwo possible solutions are really:
- switch to docker
- or we need a setup-podman action
EProblem being retrieving the latest version of Podman or building it is ugly
B(I think GitHub bringing a new distribution for action runners is much less likely) 
ENow... I could package Podman for Wolfi and we could use that
MSo how would that work?
EWe already have buildah
Xin bluebuild CI we use docker buildx with github caching and i'd 1000% recommend that to ublue as well
squashing images is the only part that doesn't work
squashing images is the only part that doesn't work
EEither we'd need to build a Buildah/Podman Wolfi container and run everything through that or I'd need a compelling justification to add it to our official image suite. We'd likely need to write our own build action
Xpodman in a container is pretty slow
MI'm currently in the docker build camp.
We're supposed to be lazy and I'm unsure maintaining another build action is preferable
We're supposed to be lazy and I'm unsure maintaining another build action is preferable
MBut my hesitation is if we cannot take advantage of BIB and zstd in future
BAs much as I appreciate these efforts. Running tools in containers in actions adds more complexity and I think that’s not the desired direction for this issue
RI might just ask in Matrix something like "Given the new Bootc containers which expects people to extend the base images with custom layers, the issues with maintaining an up-to-date Ubuntu Podman package and the incompatibilities with the older Podman version and Fedora 40 images, do you have any plans internally to support other ways of setting up Podman in GitHub Actions?"
Xi'd imagine a custom repo / ppa would be the other way
BBad words
ELet's not
BPpa is a bad word here 
Xhehe
EI agree but anything we do here is going to be some form of a tradeoff
BHonestly, right now, using doctor isn’t much of a trade-off. We lose the possible ability to use a feature. We are not yet using.
EI E. I really don't think building the latest version of Podman for deb is the solution either
