Bnumber of layers
Aostree downloads all OCI layers when it downloads our images, and then merges the data into ostree.
We could probably save a few hundred megabytes per daily system update by fixing these things.
We could probably save a few hundred megabytes per daily system update by fixing these things.
Bit doesnt download all layers
Bit just downloads updated layers, how do i know becuase thats interesting to me becuase i used to work in optimizing oci images
AWe have layers that do things like: Install 50 packages, cache RPMs in
Later layers say
But the OS still downloads both layers. So uBlue is downloading let's say 200mb of useless RPM cache files every day, because they're still inside the earlier layers due to our unoptimized Containerfiles.
/var/cache.Later layers say
delete /var/cache/*.But the OS still downloads both layers. So uBlue is downloading let's say 200mb of useless RPM cache files every day, because they're still inside the earlier layers due to our unoptimized Containerfiles.
Byes deleting caches should be a thing and it should happen in a same layer
Bin which it is created yes
AExactly, and that is what I am saying.
AAnyway that's what the tracking issue is about.
Bwell to properly optimize for wire download what would we need to do is for each package have its own run command which then goes into deleting cache or more properly just delete /var everytime becuase its actually problem to have it anyway
PThe idea being that that way a single package having a difference means you only redownload the layer with just that package, rather than a layer containing 50 other unchanged packages too, I assume
Byes
AYep, but there's one problem that brian and I talked about that we're not sure about:
- The RPM database is stored in every layer where a
- I suspect that Silverblue OCI upstream builds their layers in a smarter way with some other tools, which properly caches files separately from the RPM database?
- The RPM database is stored in every layer where a
rpm-ostree install command executes. So if anything in an earlier RPM layer updates the RPM cache, or file timestamps, we'd end up with new layers anyway.- I suspect that Silverblue OCI upstream builds their layers in a smarter way with some other tools, which properly caches files separately from the RPM database?
AThey're also working on new tools that will automatically diff files and only generate layers for modified files (I'm getting tons of emails, they're working hard at it). Hopefully F39 or F40...
Byea they are working on some better optimizations for oci builds
Phttps://youtu.be/QaKl5z6dFlM?t=209 (which got linked in #
general ) mentions "responsible chunked" images from
general ) mentions "responsible chunked" images from ostree compose image, though I'm not sure exactly how that works
Ja month ago, how did we miss this!
AAwesome, that video looks like it's about all of this. Perhaps it explains how they're solving the RPM db stuff.
AI think it requires a tool that's aware of the RPM db and generates layers like:
- 1 layer: files from X packages; NO RPM db
- 1 layer: files from Y packages; NO RPM db
- final layer: the merged RPM db
And only updating the separate layers if files from packages are changed.
- 1 layer: files from X packages; NO RPM db
- 1 layer: files from Y packages; NO RPM db
- final layer: the merged RPM db
And only updating the separate layers if files from packages are changed.
AGonna watch that whole video but not atm, still sick. 
Jyeah drink water!
AYeah got the ice water bottle here. It helps. 
JI know we'll be in a preso at some point because one of them PMed me asking if we'd be cool with it
PWas just watching it myself, I don't think it really gets too much into the details there, the bit about compose image was just mentioned in passing. But an interesting watch regardless
JI have found that every video is important because there's just too much for a single team to consume so we need every single bit of info we can get
AThat's so cool. I look forward to the explosion in popularity as we approach critical mass. 
AYeah. Ideally, someone from upstream gets interested enough to join us and help out with our infrastructure. So to achieve that, we need more growth. To get more growth, we need ads. To get alluring ads, we need to offer people something useful. To offer something useful we just need to keep doing what we're doing and getting better every day! 
JI'd like to do things like fix the broken ISOs and stuff before we deep dive into this stuff though
Abusiness speech mode
AYeah I heard the ISOs don't retry during download failures for example. :S
AIt's just a kickstart script so I dunno how we could solve that. Checking exit code and retrying a few times?
JGitHub
What happened? When using the iso to install ublue a rpm-ostree-backend error occurs mentioning invalid reference for OCI container ostree format. Relevant log output $ plasma-discover rpm-ostree-b...
Jthe installer issue is Anaconda, not us
AThat's strange,
docker://?Jhopefully all the info is in the ticket
Jcan you toss the results of your findings in the issue if you don't mind?
PSure yeah
AI see, so the installer is rewriting the URL to
docker://. But that doesn't seem to be the actual issue anyway, they quoted docs which say it's equivalent.
Bdo you need something from me on this? the main thing I know is missing was when I discovered this:
https://github.com/ublue-os/isogenerator/issues/64
I don't know if we want that sorted before announce or no
https://github.com/ublue-os/isogenerator/issues/64
I don't know if we want that sorted before announce or no
JWhere do we put the keys in this repo?
Jlike is it just inline replacing the files?
JGitHub
An action to generate custom ISOs of OCI images. Contribute to ublue-os/isogenerator development by creating an account on GitHub.
Bno, not a replace... we need both, and to import both, I can handle it, just wondering if we want it wrapped up before announce
PTo my understanding: the actual issue is ostree supports both
ostree-unverified-image:registry: and ostree-unverified-image:docker://, but discover only currently expects ostree-unverified-registry:, not anything else, so ostree-unverified-image:registry: or ostree-unverified-image:docker:// which should both be equivalent cause it to not work properly.Bback reading what does preso mean?
Jpresentation
Jyeah willing to bet this is just the discover rpm-ostree work being unfinished
Jbecause a) that's the only thing that makes sense
Jand b) we know that it's in progress, heh.
J
