N@p5 @M2 @j0rge and I had a discussion with the podman team.
We definitely want to test out zstd:chunked images in our CI. We aren't going to get immediate benefits until podman feature lands in bootc, but it would be good to get our CI prepped for it.
We definitely want to test out zstd:chunked images in our CI. We aren't going to get immediate benefits until podman feature lands in bootc, but it would be good to get our CI prepped for it.
J
Jwe need to do this for anywhere we're doing podman push
Nyeah, we'll definitely want to test this on a local registry to validate updates work as we expect.
Jyeah, or fork the bluefin repo and go
Nexactly. that works too.
Jand run some parallel test images
NYeah, actually, probably best to do this in the github runners.
RSo making an efficient zstd:chunked image is the same as making an efficient gzip image? Same concepts - many RUN commands, frequently changing files at the end etc
AAlso mind the fact that the original layers will not be recompressed probably. Only the custom ones
Nnot sure if it breaks rpm-ostree or not until we test it out.
JI don't think the order of files will make sense anymore, it tracks it at a per-file level
AUnless you add extra params
ROh, cool. So how to we make it efficient? Sorry if it's a stupid question 
Nright, we would need a zstd:chunked base image in order to take full advantage of it.
Jwe still need to wait for the fedora images themselves to be bootc and zstd:chunked, so yes, this will only be getting our CI in order.
ABut also need to make sure that rpm ostree can pull the layers
AEven if it doesn't benefit
Jso all our custom layers will be efficient, which is still a win, but when fedora is ready and we switch to their images that will be the final piece.
AOtherwise only CI
AI think buildah has a force compression that will rewrite the parent layers
Jit does
Jwe want to test that too
AIf you push tell me I wanna try pulling them see if it's faster
JI asked them in a call just now, what if the last push forces the entire thing, will it just do the entire image?
ABecause I have a sneaking suspicion that on gigabit it will be slower
Jthe answer is yes but it's not going to be as efficient as the base image coming that way was the vibe I got.
AYeah ideally you'd be building the ostree image yourself otherwise it's a bandaid
Jeither way it's a partial win, and more importantly the CI will be done, so we'll have some cycles under our belt.
AYou're still pulling 2x the files and all of the custom stuff gets invalidated and has to be rehashed
ABut I need to pull a zstd chunked image to see what's what
Jyeah IMO we should just fork a single image, the cosmic one or something, and then force compression and push zstd images
Jcosmic-zstd or whatever
Jthen we bang on that for a few days
Jhttps://github.com/castrojo/cosmic/actions/runs/9193104677 ok added it here
JI added bootc to it too
R
I think compression-format isn't in ubuntu-latest's Podman versionJGitHub
Prep in https://github.com/containers/bootc/pull/214Move pull code into deploy
WIP: Use podman pull to fetch containers
See #147 (comment)
With this bootc starts to really gain support for a diffe...
WIP: Use podman pull to fetch containers
See #147 (comment)
With this bootc starts to really gain support for a diffe...
Jwhen bootc has this we'll be able to take advantage of it, so right now we're mostly just making sure we're pushing zstd:chunked images first
Jyeah dan said we should be fine, but long term the action should just do this for everyone, like red hat's action should be the one sourcing the right version of podman, etc.
RWow! They actually acknowledged it?
Jit appears to be run by someone else, not this team
Jbuild failed, forgot to bump the runner to 24.04, trying again
RShould we squash images before chunking them? Reading this
https://www.redhat.com/sysadmin/faster-container-image-pulls
https://www.redhat.com/sysadmin/faster-container-image-pulls
RLikely not while we are using rpm-ostree, but if that's the long-term plan, that sounds cool
NI believe so. Squashing doesn't provide us any benefit currently.
Jhe squashed but I'm doing vanilla for now
Jwe should be doing all the testing with podman, rpm-ostree or bootc won't show anything different.
NRight.
J
