testing keyless cosign oidc signing

Luke Skywunker · 2024-02-19T17:37:18.644Z

I rebase to any of my images directly ```$> rpm-ostree rebase ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39 Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39 Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop:39 (digest: sha256:ec09f68334ca2fb664fab366e402f0be9313dedc1bd7b8559fb588da4af8ad70) ostree chunk layers already present: 65 custom layers already present: 1 custom layers needed: 1 (694.2 MB)```

B

bsherman•2/23/24, 7:45 AM

FYI I've done some testing of my own custom image using the PR in bluefin as a guide ( @Gerblesh @Robert (p5) you may be interested )

B

bsherman•2/23/24, 7:47 AM

Here's my findings:
1) yes, I can sign my image with OIDC / keyless
2) sadly, the PR in Bluefin probably won't actually sign successfully as it attempts to sign tags not an imageURI, but I got it working in my custom image repo
3) yes, I can directly rebase from

ublue-os/silverblue-main

ublue-os/silverblue-main

ublue-os/silverblue-main

ublue-os/silverblue-main to my oidc signed image with a URI like

ostree-image-signed:docker://ghcr.io/bsherman/silverblue-custom:latest

ostree-image-signed:docker://ghcr.io/bsherman/silverblue-custom:latest

ostree-image-signed:docker://ghcr.io/bsherman/silverblue-custom:latest

ostree-image-signed:docker://ghcr.io/bsherman/silverblue-custom:latest ... BUT!
4) because of our containers policy which has insecureAcceptAnything... it is actually doing a blind trust... I could just as easily rebase to an ancient image of mine which was not signed with OIDC

B

bsherman•2/23/24, 7:48 AM

since that policy file is part of the ublue-os main repo, it'll impact any downstream trying to do this, given a false sense of security about validating the target rebase image

B

bsherman•2/23/24, 7:48 AM

Here's an example of a rebase from stock upstream silverblue:

B

bsherman•2/23/24, 7:50 AM

First establish status

$ rpm-ostree status
State: idle
Deployments:
\u25cf fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240129.0 (2024-01-29T05:35:53Z)
               BaseCommit: bdd0afb451c1768bf872aa30c7d6ee5767f72fd75a75275a54a75f6b30804b61
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en

$ rpm-ostree status
State: idle
Deployments:
\u25cf fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240129.0 (2024-01-29T05:35:53Z)
               BaseCommit: bdd0afb451c1768bf872aa30c7d6ee5767f72fd75a75275a54a75f6b30804b61
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en

$ rpm-ostree status
State: idle
Deployments:
\u25cf fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240129.0 (2024-01-29T05:35:53Z)
               BaseCommit: bdd0afb451c1768bf872aa30c7d6ee5767f72fd75a75275a54a75f6b30804b61
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en

$ rpm-ostree status
State: idle
Deployments:
\u25cf fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240129.0 (2024-01-29T05:35:53Z)
               BaseCommit: bdd0afb451c1768bf872aa30c7d6ee5767f72fd75a75275a54a75f6b30804b61
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: langpacks-en

Now attempt to rebase to an old image with generated keypair signing

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:38-20230701
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

Now attempt to rebase to new image with OIDC signing

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

$ rpm-ostree rebase ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
Pulling manifest: ostree-image-signed:docker://ghcr.io/bsherman/silverblue-nvidia-custom:39
error: Preparing import: Fetching manifest: containers-policy.json specifies a default of `insecureAcceptAnything`; refusing usage

B

bsherman•2/23/24, 7:52 AM

Here's that first image again, failing oidc validation:

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:38-20230701
Error: no matching signatures: error verifying bundle: nil certificate provided
main.go:69: error during command execution: no matching signatures: error verifying bundle: nil certificate provided

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:38-20230701
Error: no matching signatures: error verifying bundle: nil certificate provided
main.go:69: error during command execution: no matching signatures: error verifying bundle: nil certificate provided

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:38-20230701
Error: no matching signatures: error verifying bundle: nil certificate provided
main.go:69: error during command execution: no matching signatures: error verifying bundle: nil certificate provided

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:38-20230701
Error: no matching signatures: error verifying bundle: nil certificate provided
main.go:69: error during command execution: no matching signatures: error verifying bundle: nil certificate provided

B

bsherman•2/23/24, 7:53 AM

But the second succeeds:

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:39-20240223

Verification for ghcr.io/bsherman/silverblue-custom:39-20240223 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
--- snip ---

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:39-20240223

Verification for ghcr.io/bsherman/silverblue-custom:39-20240223 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
--- snip ---

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:39-20240223

Verification for ghcr.io/bsherman/silverblue-custom:39-20240223 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
--- snip ---

$ cosign verify --certificate-oidc-issuer "https://token.actions.githubusercontent.com"     --certificate-identity-regexp "https://github.com/bsherman/ublue-custom"   ghcr.io/bsherman/silverblue-custom:39-20240223

Verification for ghcr.io/bsherman/silverblue-custom:39-20240223 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
--- snip ---

B

bsherman•2/23/24, 7:55 AM

I'm up WAY past my bedtime, but I'll look at this some more... I believe we need a policy to validate ublue-os images signed this way... which will improve our overall security, and... not give our users false sense of validation which seems to be happening at this time.

L

Luke SkywunkerOP•2/23/24, 12:53 PM

That's some really good info. Thanks for finding this out!

B

bsherman•2/23/24, 5:47 PM

I did a bit of looking for info on container policy.json and found some examples...

I haven't had a chance to test yet: https://www.mankier.com/5/containers-policy.json#Examples

containers-policy.json: syntax for the signature verification polic...

Signature verification policy files are used to specify policy, e.g.

B

bsherman•2/23/24, 6:06 PM

For reference, where I'm signing in a workflow:

https://github.com/bsherman/ublue-custom/blob/main/.github/workflows/build.yml#L181-L191

Bbsherman I did a bit of looking for info on container policy.json and found some examples...

G

Gerblesh•2/25/24, 2:16 AM

tysm for looking into this!

G

Gerblesh•2/25/24, 2:17 AM

@Luke Skywunker should we make a tracking issue for this on bluebuild?

L

Luke SkywunkerOP•2/25/24, 2:38 AM

Yes, it sounds like we'll still need to make changes to the policy.json. We'll just have to figure out some workflow on the user side to make it easier for them.

LLuke Skywunker Yes, it sounds like we'll still need to make changes to the policy.json. We'll j...

G

Gerblesh•2/25/24, 2:48 AM

I can modify the signing module to check for no pubkey and use OIDC policy.json instead of the normal one

L

Luke SkywunkerOP•2/25/24, 2:52 AM

Oh do we have a special OIDC policy.json?

LLuke Skywunker Oh do we have a special OIDC policy.json?

G

Gerblesh•2/25/24, 2:53 AM

Ehh, we could still probably just use

yq

yq

yq

yq

LLuke Skywunker Oh do we have a special OIDC policy.json?

B

bsherman•2/25/24, 2:53 AM

One is required in order to verify these images from rpm-ostree

G

Gerblesh•2/25/24, 2:54 AM

It looks pretty straightforward

G

Gerblesh•2/25/24, 2:55 AM

Judging from the reference sent

GGerblesh It looks pretty straightforward

B

bsherman•2/25/24, 2:55 AM

my eyes glazed over the other night, I was too tired

G

Gerblesh•2/25/24, 2:55 AM

And you can still directly rebase, even if it isn't verified

G

Gerblesh•2/25/24, 2:56 AM

I'm worried about non-host OIDC images though

L

Luke SkywunkerOP•2/25/24, 2:56 AM

I wonder if we could incorporate updating the policy.json using

bluebuild

bluebuild

bluebuild

bluebuild to get it to trust the image source before we start the rebase process

GGerblesh And you *can* still directly rebase, even if it isn't verified

B

bsherman•2/25/24, 2:57 AM

correct, but it's a bit of a lie, because the rpm-ostree process has a URI which indicates it has verified the image, but it has not

L

Luke SkywunkerOP•2/25/24, 2:57 AM

Since it gets baked in the image at

/usr/etc/containers/

/usr/etc/containers/

/usr/etc/containers/

/usr/etc/containers/ that means it becomes available in

/etc/containers/

/etc/containers/

/etc/containers/

/etc/containers/ and becomes "writeable"

L

Luke SkywunkerOP•2/25/24, 2:59 AM

There is a section in the policy.json for

ostree

ostree

ostree

ostree. I wonder if setting that to be strict to the user's repo would allow

rpm-ostree

rpm-ostree

rpm-ostree

rpm-ostree to properly verify the image before rebasing

Bbsherman correct, but it's a bit of a lie, because the rpm-ostree process has a URI which...

G

Gerblesh•2/25/24, 3:00 AM

Fair yeah

G

Gerblesh•2/25/24, 3:01 AM

Is it a bug then?

G

Gerblesh•2/25/24, 3:01 AM

Since it's

ostree-image-signed

ostree-image-signed

ostree-image-signed

ostree-image-signed

GGerblesh Is it a bug then?

B

bsherman•2/25/24, 3:04 AM

a bug in ublue maybe, but not in rpm-ostree itself, our

/etcc/containers/policy.json

/etcc/containers/policy.json

/etcc/containers/policy.json

/etcc/containers/policy.json forces

insecureAcceptAnything

insecureAcceptAnything

insecureAcceptAnything

insecureAcceptAnything and that's why it passes

B

bsherman•2/25/24, 3:10 AM

the way i read the example containers policy ddoc I sent... i think if ublue-os used oidc signing, we'd need something like this in our policy:

"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "fulcio": {
                        "caPath": "/path/to/fulcio_v1.crt.pem",
                        "oidcIssuer": "https://token.actions.githubusercontent.com",
                        "subjectEmail": "https://github.com/ublue-os/"
                    },
                    "rekorPublicKeyPath": "/path/to/rekor.pub",
                }
            ]

"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "fulcio": {
                        "caPath": "/path/to/fulcio_v1.crt.pem",
                        "oidcIssuer": "https://token.actions.githubusercontent.com",
                        "subjectEmail": "https://github.com/ublue-os/"
                    },
                    "rekorPublicKeyPath": "/path/to/rekor.pub",
                }
            ]

"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "fulcio": {
                        "caPath": "/path/to/fulcio_v1.crt.pem",
                        "oidcIssuer": "https://token.actions.githubusercontent.com",
                        "subjectEmail": "https://github.com/ublue-os/"
                    },
                    "rekorPublicKeyPath": "/path/to/rekor.pub",
                }
            ]

"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "fulcio": {
                        "caPath": "/path/to/fulcio_v1.crt.pem",
                        "oidcIssuer": "https://token.actions.githubusercontent.com",
                        "subjectEmail": "https://github.com/ublue-os/"
                    },
                    "rekorPublicKeyPath": "/path/to/rekor.pub",
                }
            ]

G

Gerblesh•2/25/24, 3:10 AM

"default" is "reject" though

G

Gerblesh•2/25/24, 3:11 AM

How do we get the fulcio cert?

G

Gerblesh•2/25/24, 3:11 AM

I thought cosign only needs the oidc issuer and github ref to verify

GGerblesh "default" is "reject" though

B

bsherman•2/25/24, 3:12 AM

correct... if you are starting from the default silverblue containers policy, rpm-ostree would correctly fail to rebase to

ostree-image-signed:docker://ghcr.io/ublue-os/foo-main:latest

ostree-image-signed:docker://ghcr.io/ublue-os/foo-main:latest

ostree-image-signed:docker://ghcr.io/ublue-os/foo-main:latest

ostree-image-signed:docker://ghcr.io/ublue-os/foo-main:latest regardless of type of signing

B

bsherman•2/25/24, 3:12 AM

i'm just reading the docs, i've not testedyet

B

bsherman•2/25/24, 3:17 AM

also that link I shared is bascially the same content as

man containers-policy.json

man containers-policy.json

man containers-policy.json

man containers-policy.json

GGerblesh How do we get the fulcio cert?

G

Gerblesh•2/25/24, 3:20 AM

Do you know?

B

bsherman•2/25/24, 3:25 AM

there's a URL in the doc... i haven't pursued it further

G

Gerblesh•2/25/24, 3:31 AM

Ah, here:
https://github.com/sigstore/sigstore/tree/main/pkg/tuf/repository/targets

GitHub

sigstore/pkg/tuf/repository/targets at main · sigstore/sigstore

Common go library shared across sigstore services and clients - sigstore/sigstore

G

Gerblesh•2/25/24, 4:37 AM

https://github.com/sigstore/root-signing/blob/main/repository/repository/root.json

GitHub

root-signing/repository/repository/root.json at main · sigstore/roo...

Contribute to sigstore/root-signing development by creating an account on GitHub.

G

Gerblesh•2/25/24, 7:34 AM

            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subjectEmail": "https://github.com/gerblesh"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],

            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subjectEmail": "https://github.com/gerblesh"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],

            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subjectEmail": "https://github.com/gerblesh"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],

            "ghcr.io/gerblesh/uhypr": [
                {
                "type": "sigstoreSigned",
                "fulcio": {
                      "caPath": "/etc/pki/containers/fulcio_v1.crt.pem",
                      "oidcIssuer": "https://token.actions.githubusercontent.com",
                      "subjectEmail": "https://github.com/gerblesh"
                 },
                 "rekorPublicKeyPath": "/etc/pki/containers/rekor.pub",
                 "signedIdentity": { "type": "matchRepository" }
                }
            ],

put the following into my

policy.json

policy.json

policy.json

policy.json and updating with

rpm-ostree

rpm-ostree

rpm-ostree

rpm-ostree gave me this error:

Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists

Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists

Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists

Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: A signature was required, but no signature exists

cosign verify

cosign verify

cosign verify

cosign verify is working so the image is definitely signed

G

Gerblesh•2/25/24, 7:35 AM

@Luke Skywunker got any ideas here?

G

Gerblesh•2/25/24, 7:36 AM

I have both the certs:

user@localhost ~ [1]> ls /etc/pki/containers/
 fulcio_v1.crt.pem  󰷖 rekor.pub  󰷖 ublue-os.pub

user@localhost ~ [1]> ls /etc/pki/containers/
 fulcio_v1.crt.pem  󰷖 rekor.pub  󰷖 ublue-os.pub

user@localhost ~ [1]> ls /etc/pki/containers/
 fulcio_v1.crt.pem  󰷖 rekor.pub  󰷖 ublue-os.pub

user@localhost ~ [1]> ls /etc/pki/containers/
 fulcio_v1.crt.pem  󰷖 rekor.pub  󰷖 ublue-os.pub

G

Gerblesh•2/25/24, 7:37 AM

this is all manual atm as i'm just trying to figure out what configuration actually works before I automate everything in the image build

L

Luke SkywunkerOP•2/25/24, 12:45 PM

I'm not sure. We might have to do some digging into the docs for rpm-ostree

G

Gerblesh•2/25/24, 5:28 PM

getting somewhere

G

Gerblesh•2/25/24, 5:29 PM

It now errors here:

user@localhost ~> rpm-ostree update
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Required email github.com/gerblesh/uhypr not found (got []string(nil))

user@localhost ~> rpm-ostree update
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Required email github.com/gerblesh/uhypr not found (got []string(nil))

user@localhost ~> rpm-ostree update
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Required email github.com/gerblesh/uhypr not found (got []string(nil))

user@localhost ~> rpm-ostree update
Pulling manifest: ostree-image-signed:docker://ghcr.io/gerblesh/uhypr:latest
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: Required email github.com/gerblesh/uhypr not found (got []string(nil))

G

Gerblesh•2/25/24, 5:29 PM

I needed to specify to use sigstore attachments with

container-registries.d

container-registries.d

container-registries.d

container-registries.d

testing keyless cosign oidc signing

Similar Threads

Similar Threads

Similar Threads