thanks @akdev @j0rge for the intro. for a little background, I do our open source partner program at

thanks @akdev @j0rge for the intro. for a little background, I do our open source partner program at equinix, we support a batch of CNCF projects with the help of @jeefy , and another batch of networking, operating systems, programming languages, and some low-level stuff as well.

W8EMV - EdOP•5/18/23, 7:45 PM

one big gap though is fedora-based stuff which is one reason I'm excited about this effort

WW8EMV - Ed thanks @akdev @j0rge for the intro. for a little background, I do our open sourc...

akdev•5/18/23, 7:53 PM

that's sounds super cool! I'm a SysDE at Amazon, just burn few cycles here when I can because I love open source and containers in general

WW8EMV - Ed thanks @akdev @j0rge for the intro. for a little background, I do our open sourc...

jeefy•5/18/23, 7:58 PM

lol we're all connected

akdev•5/18/23, 8:01 PM

it's very nice group of people here

Aakdev that's sounds super cool! I'm a SysDE at Amazon, just burn few cycles here when ...

pythoner6•5/18/23, 8:07 PM

Oh cool. I was an SDE at Amazon (part of the robotics org), up until the recent layoffs

akdev•5/18/23, 8:10 PM

Oh sorry to hear, I hope you land on your feet - I'm part of AFT, not as cool

pythoner6•5/18/23, 8:14 PM

Thanks! Yeah I think I'll probably be fine, just really starting to get into looking though, took a few months off to just not worry about things and finally go see some friends across the country etc.
I was technically part of AT (despite being on the east coast), got to do a bunch of infrastructure/edge device management stuff that was a lot of fun.

j0rge•5/18/23, 8:15 PM

I'm here on a transitionary "don't have a job" too, so it's a nice support group.

j0rge•5/18/23, 8:31 PM

@bsherman ok I understand what we need to do to make new keys, I have dad and gaming clan commitments tonight though

Jj0rge @bsherman ok I understand what we need to do to make new keys, I have dad and ga...

bsherman•5/18/23, 9:09 PM

Huzzah!

signer:         ublue akmods
sig_key:        17:6E:3C:E6:72:DA:64:B6:F4:27:2F:73:92:F5:A4:6F:3C:CE:86:36

signer:         ublue akmods
sig_key:        17:6E:3C:E6:72:DA:64:B6:F4:27:2F:73:92:F5:A4:6F:3C:CE:86:36

j0rge•5/18/23, 9:11 PM

I think if someone can whip up an announcement in the nvidia forum section with the link to the instructions and stuff and then we can edit/fix it up later, I have to afk for a bit

j0rge•5/18/23, 9:11 PM

or hackmd up a draft

Jj0rge I think if someone can whip up an announcement in the nvidia forum section with ...

bsherman•5/18/23, 9:12 PM

I assume that new private key is in the org-level secrets, right?

j0rge•5/18/23, 9:12 PM

yeah, same date format, I did that first.

bsherman•5/18/23, 9:16 PM

https://github.com/ublue-os/nvidia/pull/104

GitHub

chore: update akmods pubkey with nicer CN by bsherman · Pull Reques...

Jorge regenerated the new akmods key to include meaningful CN, "ublue akmods", instead of the default "akmods local singing CA".
For #100

Arcitec•5/18/23, 10:26 PM

Sorry if anyone's tried to contact me. I'm a bit sick at the moment. You gave me a laugh with this though:

instead of the default "akmods local singing CA".

Who was doing the singing? Jorge?

Arcitec•5/18/23, 10:39 PM

I've also discovered that we can shrink our OCI downloads. Possibly by quite a lot. I can't do anything at the moment, too tired. But I committed some earlier work that takes care of that for startingpointstartingpoint and added to the board to track it for the other repos:

https://github.com/orgs/ublue-os/projects/1/views/1?pane=issue&itemId=28591278

j0rge•5/18/23, 11:30 PM

I was under the impression that we wanted more layers not less

j0rge•5/18/23, 11:33 PM

@bsherman ok I'm back now and can work on announcement stuff

AArcitec I've also discovered that we can shrink our OCI downloads. Possibly by quite a l...

Bigpod•5/18/23, 11:33 PM

yes and no, first of all standard oci ideas dont work here been there tried that

Bigpod•5/18/23, 11:34 PM

first of all more layers more better for us, unlike with general OCI usage where preferably you would have as little layers possible even just one

Bigpod•5/18/23, 11:35 PM

also we create 10 layers on top of about 50 layers that come from our base image

Bigpod•5/18/23, 11:37 PM

next fun fact only way to properly ensure size doesnt grow from caches and such is to delete them during same run command as creates them as in some (but not all) cases as soon as that run command ends that layer is set in stone and doesnt realy change

CyberGlitterhoof•5/18/23, 11:37 PM

yeah deleting caches in the same command makes sense since it creates diffs that force new downloads for those layers

CyberGlitterhoof•5/18/23, 11:39 PM

bvut I think j0rge is right you want more layers in general since there's more chances to have duplicate layers that you can skip downloading after the first image is downloaded

CCyberGlitterhoof bvut I think j0rge is right you want more layers in general since there's more c...

Bigpod•5/18/23, 11:39 PM

in our case yes in general oci not realy

Bigpod•5/18/23, 11:40 PM

in general OCI you want to have as little layers as possible with most commonality in your stack, but we are not doing services we are doing os where certien things dont matter

Bigpod•5/18/23, 11:43 PM

also i dont think those caches are actually costing us that much

j0rge•5/18/23, 11:46 PM

but either way we'd need to wait to do what their team recommends as far as layers go

Jj0rge I was under the impression that we wanted more layers not less

Arcitec•5/18/23, 11:47 PM

More layers for actual installed packages, yes. Splitting packages is a good thing.

What we are doing right now is not a good thing.

We are currently having:

- Totally useless layers (like mkdir somethingmkdir something).
- Layers full of RPM packages and cache files and temp files. Probably shipping like 200mb of useless data on every uBlue image at the moment.

AArcitec More layers for actual installed packages, yes. Splitting packages is a good thi...

Bigpod•5/18/23, 11:47 PM

layers dont matter

Bigpod•5/18/23, 11:47 PM

number of layers

BBigpod layers dont matter

Arcitec•5/18/23, 11:48 PM

ostree downloads all OCI layers when it downloads our images, and then merges the data into ostree.

We could probably save a few hundred megabytes per daily system update by fixing these things.

Bigpod•5/18/23, 11:48 PM

it doesnt download all layers

Bigpod•5/18/23, 11:49 PM

it just downloads updated layers, how do i know becuase thats interesting to me becuase i used to work in optimizing oci images

Arcitec•5/18/23, 11:49 PM

We have layers that do things like: Install 50 packages, cache RPMs in /var/cache/var/cache.

Later layers say delete /var/cache/*delete /var/cache/*.

But the OS still downloads both layers. So uBlue is downloading let's say 200mb of useless RPM cache files every day, because they're still inside the earlier layers due to our unoptimized Containerfiles.

Bigpod•5/18/23, 11:49 PM

yes deleting caches should be a thing and it should happen in a same layer

Bigpod•5/18/23, 11:50 PM

in which it is created yes

Arcitec•5/18/23, 11:50 PM

Exactly, and that is what I am saying.

Arcitec•5/18/23, 11:50 PM

Anyway that's what the tracking issue is about.

Bigpod•5/18/23, 11:53 PM

well to properly optimize for wire download what would we need to do is for each package have its own run command which then goes into deleting cache or more properly just delete /var everytime becuase its actually problem to have it anyway

BBigpod well to properly optimize for wire download what would we need to do is for each...

pythoner6•5/18/23, 11:54 PM

The idea being that that way a single package having a difference means you only redownload the layer with just that package, rather than a layer containing 50 other unchanged packages too, I assume

Ppythoner6 The idea being that that way a single package having a difference means you only...

Bigpod•5/18/23, 11:54 PM

yes

Arcitec•5/18/23, 11:55 PM

Yep, but there's one problem that brian and I talked about that we're not sure about:

- The RPM database is stored in every layer where a

rpm-ostree install

rpm-ostree install

command executes. So if anything in an earlier RPM layer updates the RPM cache, or file timestamps, we'd end up with new layers anyway.

- I suspect that Silverblue OCI upstream builds their layers in a smarter way with some other tools, which properly caches files separately from the RPM database?

Arcitec•5/18/23, 11:55 PM

They're also working on new tools that will automatically diff files and only generate layers for modified files (I'm getting tons of emails, they're working hard at it). Hopefully F39 or F40...

Bigpod•5/18/23, 11:56 PM

yea they are working on some better optimizations for oci builds

pythoner6•5/18/23, 11:57 PM

https://youtu.be/QaKl5z6dFlM?t=209 (which got linked in #

general ) mentions "responsible chunked" images from ostree compose imageostree compose image, though I'm not sure exactly how that works

j0rge•5/18/23, 11:58 PM

a month ago, how did we miss this!

Arcitec•5/18/23, 11:58 PM

Awesome, that video looks like it's about all of this. Perhaps it explains how they're solving the RPM db stuff.

Arcitec•5/18/23, 11:59 PM

I think it requires a tool that's aware of the RPM db and generates layers like:

- 1 layer: files from X packages; NO RPM db
- 1 layer: files from Y packages; NO RPM db
- final layer: the merged RPM db

And only updating the separate layers if files from packages are changed.

thanks @akdev @j0rge for the intro. for a little background, I do our open source partner program at

Similar Threads

Similar Threads

Similar Threads