ZSTD Chunked Images

Original message was deleted

Noel•5/22/24, 1:58 PM

@p5 @M2 @j0rge and I had a discussion with the podman team.

We definitely want to test out zstd:chunked images in our CI. We aren't going to get immediate benefits until podman feature lands in bootc, but it would be good to get our CI prepped for it.

j0rge•5/22/24, 1:59 PM

j0rge•5/22/24, 1:59 PM

we need to do this for anywhere we're doing podman push

Noel•5/22/24, 2:00 PM

yeah, we'll definitely want to test this on a local registry to validate updates work as we expect.

j0rge•5/22/24, 2:00 PM

yeah, or fork the bluefin repo and go

Noel•5/22/24, 2:00 PM

exactly. that works too.

j0rge•5/22/24, 2:00 PM

and run some parallel test images

Noel•5/22/24, 2:00 PM

Yeah, actually, probably best to do this in the github runners.

Robert•5/22/24, 2:01 PM

So making an efficient zstd:chunked image is the same as making an efficient gzip image? Same concepts - many RUN commands, frequently changing files at the end etc

antheas•5/22/24, 2:01 PM

Also mind the fact that the original layers will not be recompressed probably. Only the custom ones

Noel•5/22/24, 2:01 PM

not sure if it breaks rpm-ostree or not until we test it out.

j0rge•5/22/24, 2:01 PM

I don't think the order of files will make sense anymore, it tracks it at a per-file level

antheas•5/22/24, 2:01 PM

Unless you add extra params

Jj0rge I don't think the order of files will make sense anymore, it tracks it at a per-...

Robert•5/22/24, 2:02 PM

Oh, cool. So how to we make it efficient? Sorry if it's a stupid question

Aantheas Also mind the fact that the original layers will not be recompressed probably. O...

Noel•5/22/24, 2:02 PM

right, we would need a zstd:chunked base image in order to take full advantage of it.

j0rge•5/22/24, 2:02 PM

we still need to wait for the fedora images themselves to be bootc and zstd:chunked, so yes, this will only be getting our CI in order.

antheas•5/22/24, 2:02 PM

But also need to make sure that rpm ostree can pull the layers

antheas•5/22/24, 2:02 PM

Even if it doesn't benefit

j0rge•5/22/24, 2:03 PM

so all our custom layers will be efficient, which is still a win, but when fedora is ready and we switch to their images that will be the final piece.

antheas•5/22/24, 2:03 PM

Otherwise only CI

antheas•5/22/24, 2:03 PM

I think buildah has a force compression that will rewrite the parent layers

j0rge•5/22/24, 2:03 PM

it does

j0rge•5/22/24, 2:03 PM

we want to test that too

antheas•5/22/24, 2:04 PM

If you push tell me I wanna try pulling them see if it's faster

j0rge•5/22/24, 2:04 PM

I asked them in a call just now, what if the last push forces the entire thing, will it just do the entire image?

antheas•5/22/24, 2:04 PM

Because I have a sneaking suspicion that on gigabit it will be slower

j0rge•5/22/24, 2:04 PM

the answer is yes but it's not going to be as efficient as the base image coming that way was the vibe I got.

antheas•5/22/24, 2:05 PM

Yeah ideally you'd be building the ostree image yourself otherwise it's a bandaid

j0rge•5/22/24, 2:05 PM

either way it's a partial win, and more importantly the CI will be done, so we'll have some cycles under our belt.

antheas•5/22/24, 2:05 PM

You're still pulling 2x the files and all of the custom stuff gets invalidated and has to be rehashed

antheas•5/22/24, 2:06 PM

But I need to pull a zstd chunked image to see what's what

j0rge•5/22/24, 2:06 PM

yeah IMO we should just fork a single image, the cosmic one or something, and then force compression and push zstd images

j0rge•5/22/24, 2:06 PM

cosmic-zstd or whatever

j0rge•5/22/24, 2:06 PM

then we bang on that for a few days

j0rge•5/22/24, 2:22 PM

https://github.com/castrojo/cosmic/actions/runs/9193104677 ok added it here

j0rge•5/22/24, 2:24 PM

I added bootc to it too

Robert•5/22/24, 2:25 PM

I think compression-format isn't in ubuntu-latest's Podman version

j0rge•5/22/24, 2:27 PM

https://github.com/containers/bootc/pull/215

GitHub

WIP: Use `podman pull` to fetch containers by cgwalters · Pull Requ...

Prep in https://github.com/containers/bootc/pull/214Move pull code into deploy

WIP: Use podman pull to fetch containers
See #147 (comment)
With this bootc starts to really gain support for a diffe...

j0rge•5/22/24, 2:27 PM

when bootc has this we'll be able to take advantage of it, so right now we're mostly just making sure we're pushing zstd:chunked images first

RRobert 😆 I think compression-format isn't in ubuntu-latest's Podman version

j0rge•5/22/24, 2:28 PM

yeah dan said we should be fine, but long term the action should just do this for everyone, like red hat's action should be the one sourcing the right version of podman, etc.

Robert•5/22/24, 2:29 PM

Wow! They actually acknowledged it?

j0rge•5/22/24, 2:29 PM

it appears to be run by someone else, not this team

j0rge•5/22/24, 2:29 PM

build failed, forgot to bump the runner to 24.04, trying again

Robert•5/22/24, 2:33 PM

Should we squash images before chunking them? Reading this
https://www.redhat.com/sysadmin/faster-container-image-pulls

$ buildah bud --squash --format oci -t example.com/my-new-zstd-chunked-image
$ buildah push --compression-format zstd:chunked example.com/my-new-zstd-chunked-image

$ buildah bud --squash --format oci -t example.com/my-new-zstd-chunked-image
$ buildah push --compression-format zstd:chunked example.com/my-new-zstd-chunked-image

Robert•5/22/24, 2:34 PM

Likely not while we are using rpm-ostree, but if that's the long-term plan, that sounds cool

RRobert Should we squash images before chunking them? Reading this https://www.redhat.c...

Noel•5/22/24, 2:35 PM

I believe so. Squashing doesn't provide us any benefit currently.

j0rge•5/22/24, 2:35 PM

he squashed but I'm doing vanilla for now

j0rge•5/22/24, 2:35 PM

we should be doing all the testing with podman, rpm-ostree or bootc won't show anything different.

Jj0rge we should be doing all the testing with podman, rpm-ostree or bootc won't show a...

Noel•5/22/24, 2:36 PM

Right.

j0rge•5/22/24, 2:36 PM

ZSTD Chunked Images

Similar Threads

Similar Threads

Similar Threads