Labels and metadata
Hello! I am looking to add additional Lables to my container and overwrite labels that are coming from the template in bluebuild-cli https://github.com/blue-build/cli/blob/main/template/templates/Containerfile.j2. It adds
I would like to set this to my own readme along with the rest of the artifacthub labels as outlined in this open issue: https://github.com/blue-build/cli/issues/342.
I would add labels using a containerfile snippet, but there is this note "# Labels are added last since they cause cache misses with buildah" and I want to overwrite the label above.
In general, is there a mechanism to facilitate added labels to the end of the generated containerfile? There seemed to be some discussion of this here: https://github.com/blue-build/cli/issues/180
I also looked at trying to integrate this into the build pipeline: https://github.com/ublue-os/image-template/blob/2e57cc55652e4ebe2c6a134212f542c74d4ef6c6/.github/workflows/build.yml#L77C1-L106C31 But I didn't see a way to inject it into the bluebuild ci pipeline.
Any direction would be greatly appreciated! Thanks!
Any direction would be greatly appreciated! Thanks!
GitHub
blue-build/cli
BlueBuild's command line program that builds custom Fedora Atomic images based on your recipe.yml - blue-build/cli
GitHub
arriving at a stable, functional, and sensical recipe standard · I...
(this issue is filed under CLI, because this is the only place changes would have to be made to implement changes to the recipe format) I had an epiphany. But let me preface a bit; if the purpose o...
GitHub
image-template/.github/workflows/build.yml at 2e57cc55652e4ebe2c6a1...
Build your own custom Universal Blue Image! Contribute to ublue-os/image-template development by creating an account on GitHub.
31 Replies
There is currently not a way to set that, but I think this would be a pretty cool feature to add
I'm been exploring the codebase for the bluebuild-cli and it looks doable, but I found some considerations. The current current structure could be updated to handle the labels in a fairly straightforward way except for when it comes to rechunking. Since the rechunking removes the labels and they have to be reapplied, I looked at a few routes. I had hoped to inspect the image created before rechunking and pull out the labels for generic reapplication; however, there are a lot of labels in there we don't care about, including some related to the rechunking process. The current rechunking code manually builds back the defined set of labels and passes them into rechunking.
In order to support custom labeling, we would need to have access to those labels at the point in the code, but this information is no longer available. So I looked into the containerfile generation code and the build code. Please let me know if I'm missing anything in the current build outline: 1. par_iter try_for_each over the recipe paths generates the containerfiles in a temp directory after parsing the recipe and gathering some data from the Driver https://github.com/blue-build/cli/blob/a3cd3a07f0f149141ce7d343ff0bc93b61aedca8/src/commands/build.rs#L198C1-L211C13 2. another par_iter try_fold operates over the recipe paths to produce image strings and in each of these the recipe is reparsed and the actual build is performed, including rechunking. https://github.com/blue-build/cli/blob/a3cd3a07f0f149141ce7d343ff0bc93b61aedca8/src/commands/build.rs#L223C1-L236C17 In order to have access to the label information to reapply at rechunking, I think we would need access to the recipe information and the meta-data.
In order to support custom labeling, we would need to have access to those labels at the point in the code, but this information is no longer available. So I looked into the containerfile generation code and the build code. Please let me know if I'm missing anything in the current build outline: 1. par_iter try_for_each over the recipe paths generates the containerfiles in a temp directory after parsing the recipe and gathering some data from the Driver https://github.com/blue-build/cli/blob/a3cd3a07f0f149141ce7d343ff0bc93b61aedca8/src/commands/build.rs#L198C1-L211C13 2. another par_iter try_fold operates over the recipe paths to produce image strings and in each of these the recipe is reparsed and the actual build is performed, including rechunking. https://github.com/blue-build/cli/blob/a3cd3a07f0f149141ce7d343ff0bc93b61aedca8/src/commands/build.rs#L223C1-L236C17 In order to have access to the label information to reapply at rechunking, I think we would need access to the recipe information and the meta-data.
GitHub
cli/src/commands/build.rs at a3cd3a07f0f149141ce7d343ff0bc93b61aedc...
BlueBuild's command line program that builds custom Fedora Atomic images based on your recipe.yml - blue-build/cli
I see 2 kinds of labeling, the custom labeling and the built-in labeling (which uses the metadata from the driver. I think it might make sense to have labeling built at the time the recipe is parse and the container file constructed and then pass this data out for use in the build process. In this way, we avoid redundant parsing and driver calls. I was also thinking that it would be good to warn in the event that a custom label conflicts with a built-in label. This could all be done at the point of generation.
There is a slight complication with passing this data around given the par_iter around generation followed by a par_iter to build the images. I was playing around with this and was having trouble getting the data passed out (I didn't dig into it too much yet), but the issue is removed if both generation and build happen within the same par_iter. I'm sure both paths are doable though.
What are your thoughts? I'm happy to work on this feature. It would be a more general solution to the artifacthub issue that is currently open and all for that to be easily implemented. This would require some updates to the recipe schema. The are a lot of options here, depending upon how you want to evolve it.
What are your thoughts? I'm happy to work on this feature. It would be a more general solution to the artifacthub issue that is currently open and all for that to be easily implemented. This would require some updates to the recipe schema. The are a lot of options here, depending upon how you want to evolve it.
Nice catch, now I'm sitting here thinking why I even did that in the first place lol. Might be vestigial from
v0.8
If you think you have an idea of how to handle this feature, go for it!As for updating the schema, you'd want to look at https://github.com/blue-build/schema/blob/main/src-tsp/main.tsp
GitHub
schema/src-tsp/main.tsp at main · blue-build/schema
TypeSpec schema for BlueBuild modules and recipes. Contribute to blue-build/schema development by creating an account on GitHub.
I have definitely had this happen myself. 😅 lol.
OK, will do! I will iterate and open an issue in github and make a PR. Thanks for this awesome project BTW!
Let me know if you have any questions. The tests should be setup to work for forks
Perfect, will do!
Oh about the recipe parsing, it's cached with an internal function
So calling it again with the same path will just return a clone of the computed recipe
Ahh, OK. I missed that. Are the Driver queries that get various bits of metadata also cahced? I should look more closely at the caching structure.
They can be. You can always call a
_no_cache version of those functions. I make use of the cached crateI like this, I think I can create a structure that will hold all of the metadata necessary to compose all aspects of the labels using the same cache approach as the recipe parse. Then we can leave the current par_iter structure in place. I was thinking that it is nice that the first iteration does validation and creation of the containerfiles prior to any build. It would error out and force full validation before even starting the builds of any recipes. I think this makes sense so it would be nice to keep that. With the cache we can build everything needed in the generation command but leverage it from the cache in the build command. I will explore this approach. Thanks for the information!
No problem!
I am getting ready to update the schema for the recipe. I was going to add a new top level key called "additional-labels" that would be an optional array of objects so that the yaml would be something like:
This is the most compact, but we could also do an array of name, value pairs intead. Do you have a preference?
I think since labels are key/value pairs that a map like what you have is perfect
Though we could just call it labels
I considered "labels" as well and I think that would be fine. l went with "additional-labels" only because it felt more explicit since there are built-in labels already provided. But it isn't a huge point for me. Do you still prefer "labels"? If so, I will make that change.
Yeah let's go with "labels". Short, sweet, and to the point
I'm working on testing everything now. I am trying to test schema changes locally. It isn't working out as I had hoped. Do you have a suggested way of testing schema changes to ensure that validation is working properly? I'm new to creating these kinds of schemas and working with them. I can build the new schema and I tried referencing them by file for testing, but that just highlights everything as invalid, so I think I'm missing something...
You can use
--skip-validation to ignore schema issues
There isn't really a way to set that up locallyThat is what I've been doing, just wasn't sure if there was a way to test it directly.
ok, I should have a PR soon for both the schema and the labels feature. I have been manually validating cases. I'm going to validate against my use case as well.
I do have a question about the automated tets. I didn't see in the integration tests how it might check the output for the labels. I can add them to the some of the yamls involved in the test cases. Are these just functional tests to make sure the workflow works? I am going to do some unit tests for the variations, just wasn't sure if we should/could do some validation tests like yes labels ended in the containerfile and yes the rechunked final image has all of the labels post rechunking.
I do have a question about the automated tets. I didn't see in the integration tests how it might check the output for the labels. I can add them to the some of the yamls involved in the test cases. Are these just functional tests to make sure the workflow works? I am going to do some unit tests for the variations, just wasn't sure if we should/could do some validation tests like yes labels ended in the containerfile and yes the rechunked final image has all of the labels post rechunking.
Yeah if you want to add those, that would be good. The
justfile is used to run the various tests for the recipes in ./integration-test/test-repoyes, I was using those to test, but I was manually inspecting output. I don't see any automated inspected post processing cases. Am I missing something? Sorry if I missed something obvious.
Yeah I haven't set that up at all
Those tests were setup for testing the workflow
ok. I will just take care of the unit tests for now then. I'm not sure of the best design for testing post workflow. I could write shell scripts for this, but my experience with too many shell script is that they tend to get fragmented and unmaintainable over time. But I'm not sure how else to do it. Is this kind of test something you would like to invest in? I noticed that you are using nushell in several places. I haven't worked with it yet, but it does seem to be more consistent/maintainable than a bunch of standard shell scripts. If this is something you would like to have, I'm happy to invest some time in this approach.
It's fine, that's something that I can try to pick up later
Don't want to dump a bunch of work on you
No worries! I'm happy to help. I was using the image template from ublue for a custom image and came across this project and liked the approach more. Some of the items in that template just don't function despite saying you can just uncomment them (the rechunking is like this). It felt like a rather disparate way of shoving a bunch of things together. I very much appreciate the recipe workflow for its clarity. I also liked the fact that rechunking just worked when I tested it. It feels more "solid" if that makes sense. So I'm happy to contribute back as I will be maintaining and enhancing my own custom image. I also like working on rust code, so that's a plus in my mind. 😁
We're more than happy to have others willing to contribute! Glad you're enjoying the project
GitHub
Feature custom container labels by Zyell · Pull Request #578 · bl...
This is the feature as discussed in discord: https://discord.com/channels/1198292321037455360/1428080454417776822
The goal is to allow custom labels to be declared in the recipe and applied to the ...
CLI is the most demanding part of BlueBuild for maintaining, so I'm glad that you want to contribute
I'll take a look this weekend!
Happy to help!