RunPod•4w ago

Registry auth credential not working

Hi all, I have workflow automation that builds a container, adds it to my container registry (ghcr), and then tries to spin up a pod using this container via https://rest.runpod.io/v1/pods. I pass in the ID of the Container Registry Auth credential I created in RunPod for my container registry. While I can use my container registry credentials to pull the container by hand without issue, RunPod can't. The pod's logs show authorization errors. I've triple checked the container registry auth credential ID, as well as the username (mine, as a org github admin) and PAT stored there (which has read:packages and read:org permissions). I also tried using _json_key_base64 as the username, as suggested by the AI bot support, but that didn't work. When I filed a ticket with support, they recommended I move to using templates, which won't work because I need to mount a network volume and templates don't support that. Any thoughts?

23 Replies

Jason•4w ago

use templates, mount it in your requests

billpressOP•4w ago

Got it, so I can mount it after standing up the pod as a separate request. Is that right? Just curious: is the registry auth credentials known not to work? Or does it sound like I'm doing something wrong?

Jason•4w ago

Before* yes, in the create pod request it does, it is set up correctly just add the template-related things in your request body to use the template and leave everything as it is i think ( if you use ns)

billpressOP•4w ago

I get errors using the networkVolumeId and volumeMountPath parameters when I specify a templateId. I believe those parameters are unsupported with the v1 API when specifying a templateId.

Jason•4w ago

Hmm try to send this back to your ticket

billpressOP•4w ago

What is the correct username for my registry auth credential? Is it my username (as an admin), the org name, or a string literal as suggested by the AI support? Though none of them worked for me, I'd like to know which I should be using as I work with support to debug the issue.

Dj•4w ago

For GHCR, its your username in all lowercase and a Personal Access Token with read:packages for Pulling or repo and write:packages for Publishing. PAT's are the "Tokens (classic)" option on this page: https://github.com/settings/tokens

billpressOP•4w ago

Thanks, @Dj . That matches what I was doing (my username, all lowercase, and a classic token). Only pulling, not publishing, so I have read:packages as well as read:org for good measure. Sadly, it does not work (and I've recreated the token and rechecked/recreated things multiple times).

Dj•4w ago

I just tested using the rest api to do the same thing, so I'm certain its not us. Can you show me the request you're making? None of the ids in the request can be used by anyone but you, so they're safe to share. "to do the same thing" - I did POST https://rest.runpod.io/v1/pods with the body:

{
  "allowedCudaVersions": [
    "12.4", "12.5", "12.6", "12.7", "12.8"
  ],
  "cloudType": "SECURE",
  "computeType": "GPU",
  "containerRegistryAuthId": "clzdaifot0001l90809257ynb",
  "dataCenterIds": [ "US-IL-1" ],
  "gpuCount": 1,
  "gpuTypeIds": [ "NVIDIA GeForce RTX 4090" ],
  "gpuTypePriority": "availability",
  "imageName": "ghcr.io/partyhatgg/runpod-faster-whisper:main",
  "name": "my api pod",
  "networkVolumeId": "folpc6loeb",
  "ports": [
    "8888/http",
    "22/tcp"
  ]
}

{
  "allowedCudaVersions": [
    "12.4", "12.5", "12.6", "12.7", "12.8"
  ],
  "cloudType": "SECURE",
  "computeType": "GPU",
  "containerRegistryAuthId": "clzdaifot0001l90809257ynb",
  "dataCenterIds": [ "US-IL-1" ],
  "gpuCount": 1,
  "gpuTypeIds": [ "NVIDIA GeForce RTX 4090" ],
  "gpuTypePriority": "availability",
  "imageName": "ghcr.io/partyhatgg/runpod-faster-whisper:main",
  "name": "my api pod",
  "networkVolumeId": "folpc6loeb",
  "ports": [
    "8888/http",
    "22/tcp"
  ]
}

Dj•4w ago

Here's where you can get both ids:

Dj•4w ago

Ignore that this run has 0 GPUs, I stopped the Pod before I realized I should get a screenshot and got a CPU instance >.>

billpressOP•4w ago

Thank you, @Dj . I was getting the IDs from the correct places (i.e., where you showed them). I'm using the body:

{
  "gpuTypeIds": [ "NVIDIA H200" ],
  "imageName": "ghcr.io/foo/bar/bar:latest",
  "name": "gh-action-pod",
  "containerDiskInGb": 20,
  "containerRegistryAuthId": "cma95l3640001jr08778nvk4g",
  "networkVolumeId": "2ak38tc1dr",
  "volumeMountPath": "/workspace",
  "ports": [ "8888/http" ],
}

{
  "gpuTypeIds": [ "NVIDIA H200" ],
  "imageName": "ghcr.io/foo/bar/bar:latest",
  "name": "gh-action-pod",
  "containerDiskInGb": 20,
  "containerRegistryAuthId": "cma95l3640001jr08778nvk4g",
  "networkVolumeId": "2ak38tc1dr",
  "volumeMountPath": "/workspace",
  "ports": [ "8888/http" ],
}

The pod gets provisioned successfully and in the same data center as the network volume, but the pod logs say:

error pulling image: Error response from daemon: Head "https://ghcr.io/v2/foo/bar/bar/manifests/latest": unauthorized
error creating container: unauthorized to use image ghcr.io/foo/bar/bar:latest
create container ghcr.io/foo/bar/bar:latest
error creating container: container: create: container create: Error response from daemon: No such image: ghcr.io/foo/bar/bar:latest

error pulling image: Error response from daemon: Head "https://ghcr.io/v2/foo/bar/bar/manifests/latest": unauthorized
error creating container: unauthorized to use image ghcr.io/foo/bar/bar:latest
create container ghcr.io/foo/bar/bar:latest
error creating container: container: create: container create: Error response from daemon: No such image: ghcr.io/foo/bar/bar:latest

With the last two lines repeated periodically.

Dj•4w ago

This: Head "https://ghcr.io/v2/foo/bar/bar/manifests/latest": unauthorized Is a GitHub error. Are you running this from a GitHub action? You need to do something similiar to this, but with packages: read to be able to frward your GITHUB_TOKEN to RunPod.

billpressOP•4w ago

I am running this from a github workflow. I added these permissions to my workflow and was hopeful (I didn' tknow about these). Unfortunately, I got the same result.

Dj•4w ago

Tell me about the workflow, can I assume you're making a new token with the gql api, adding it to your runpod account and then attempting a deploy with your newly created container auth id?

billpressOP•4w ago

No, I set the container auth id by hand in the runpod UI. I have it stored as a secret in Github (though I guess I now know I can just use a variable). The workflow just gets the value of the secret to populate the runpod request body.

Dj•4w ago

I think the issue is we GitHub regens that container id. You could use the GraphQL API to fully automate this flow? Nevermind, the REST API can do the whole thing. Nevermind again I misinterpreted how you're setting the auth ids. Are you using the github.actor as a username and your GitHub PAT as the password? I'm not 100% certain what your setup is, but my mindset would be to just dynamically do all of this from the context of the action. Then the action is reusable for others as well.

permissions:
    packages: read

permissions:
    packages: read

POST https://rest.runpod.io/v1/containerregistryauth { "name": "GitHub Action", "username": "$github.actor", "password": "$secrets.GITHUB_TOKEN" } POST /v1/pods ... DEL /v1/containerregistryauth/:id Where those $ variables in strings are interpolated from the actions runner. I think it could also be useful for us (RunPod) to build this first party into our test-runner. It's a bit of a mess, so remaking it with additional functionality like this may help.

billpressOP•4w ago

Interesting. I had created my own PAT with read:packages and used my username in the container registry auth (that I created using the RunPod UI). I like your flow better, though, creating it dynamically, but isn't the token associated with my username, since it's my PAT? Will it work with $github.actor when somebody else is running the workflow? I know we're well into GitHub help, vs RunPod help. I really appreciate your support here. I misunderstood. This makes a lot of sense. I'll try it now. @Dj I successfully created the container registry auth token from my workflow in the way you recommended (and see it in the portal before it's deleted). Unfortunately, I still continue to get the same error messages in the pod logs. Note that I also set the following now at the top of my workflow job:

  permissions:
    contents: read
    packages: write
    id-token: write

  permissions:
    contents: read
    packages: write
    id-token: write

billpressOP•4w ago

@Dj Here's my complete custom action, if it's helpful to see exactly what I'm doing. Thank you again for all your help.

action.yml

billpressOP•3w ago

Hi @Dj , I never heard back from you about this. I eventually got the template-based authentication to work; however, I was never able to get logging in with a container registry auth ID (permanent or dynamically created) to work. I don't love using a template, but it'll do for now. Anyway, I appreciated your help here. It's really great to have somebody from RunPod interacting with customers in this way. Thank you.

Dj•3w ago

Hey, sorry I never got to respond to you. We recently solved some issues related to how we pull images with a user credential. So maybeee it'll work now (👀?) - but I am happy to help and I'm glad I make your RunPod experience just a little better even if things didn't work out the way you intended ❤️

billpressOP•3w ago

Thanks, DJ. I'm stuck on a different issue at the moment. I'll post a separate topic just to keep it clean... I'd love your thoughts on this, @Dj , if you're up for giving it a quick look.

Gaming

Programming

Registry auth credential not working

Did you find this page helpful?