Runpod•8mo ago

Container images loading stuck in a loop when launching

I have workers that never actually launch after pulling containers. Have no idea how to debug this. Deleted and recreated the endpoint and get the same behavior. Any thoughts on how to resolve? It is extra aggrevating as I've had to spin this up because of the EU-SE-1 performance degradation and now getting hit with this issue.

endpoint id: d9b5s5qpbl0sfb

=== Snip from the logs. This is just looping repeatedly. Have the credentials set for dockerhub, the image is published and available, etc. ===

2025-05-07T19:34:02Z loading container image from cache
2025-05-07T19:34:50Z Loaded image: docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z 0.1.24-dev0 Pulling from docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z Digest: sha256:b066e7235b92701dca45b26a3da6437e1fdc3ca96f751fd5bd614cdb40f532bb
2025-05-07T19:34:51Z Status: Image is up to date for docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z worker is ready
2025-05-07T19:37:23Z create container docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:23Z loading container image from cache
2025-05-07T19:37:31Z create container: still fetching image docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:32Z create container docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:32Z create container: still fetching image docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:33Z Loaded image: docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z 0.1.24-dev0 Pulling from docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z Digest: sha256:b066e7235b92701dca45b26a3da6437e1fdc3ca96f751fd5bd614cdb40f532bb
2025-05-07T19:37:34Z Status: Image is up to date for docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z worker is ready

David•5/7/25, 7:58 PM

Yep same here

Mandulis - Max - 2/2•5/7/25, 8:15 PM

We also experience problems

Mandulis - Max - 2/2•5/7/25, 8:16 PM

Workers are running and we pay but they only stay in the queue and don't get processed...

zongheng1619•5/7/25, 8:44 PM

same here.. I have stopped all of my workers, else it's charging me all the time

Zzongheng1619 same here.. I have stopped all of my workers, else it's charging me all the time

Mandulis - Flix•5/7/25, 8:45 PM

Same for us.

Anubhav•5/7/25, 8:58 PM

I am facing the same. Do you know how can we get this complaint escalated?

noahpantspartyOP•5/7/25, 9:06 PM

I'll tag @Dj to see if they can provide any insight to what is going on. Sounds like it is impacting quite a few of us.

Dj•5/7/25, 9:06 PM

Great

Dj•5/7/25, 9:06 PM

Okay one sec

Dj•5/7/25, 9:06 PM

I thought this was isolated to like 2-3 people

Dj•5/7/25, 9:07 PM

Working on this, let me make a lot more noise

Dj•5/7/25, 9:11 PM

Can I ask, are these public or private images?

noahpantspartyOP•5/7/25, 9:11 PM

in my case, this is/was happening with a private image

Anubhav•5/7/25, 9:12 PM

In my case public.

Dj•5/7/25, 9:12 PM

GHCR or Docker Hub?

Anubhav•5/7/25, 9:12 PM

DockerHub for me,.

noahpantspartyOP•5/7/25, 9:12 PM

DockerHub

georg•5/7/25, 9:19 PM

DockerHub for me with a private image.

Dj•5/7/25, 9:21 PM

I'm working on getting this escalated, I can definitely see the pattern we're just isolating the problem

Ggeorg DockerHub for me with a private image.

Mandulis - Flix•5/7/25, 9:32 PM

Same

Dj•5/7/25, 9:41 PM

Small update, this is being looked into.

Nnoahpantsparty I have workers that never actually launch after pulling containers. Have no idea...

Dj•5/7/25, 9:44 PM

Can you check your Container Registry Token you provided RunPod to ensure its valid?

Dj•5/7/25, 9:44 PM

You can check on Docker Hub directly to avoid messing with anything.

https://app.docker.com/settings/personal-access-tokens

georg•5/7/25, 9:46 PM

We can release new endpoint with the same token. nothing change and it is working.

DDj Can you check your Container Registry Token you provided RunPod to ensure its va...

Mandulis - Flix•5/7/25, 9:48 PM

Yes, valid.

Anubhav•5/7/25, 9:55 PM

It works for me now.

DDj Can I ask, are these public or private images?

David•5/7/25, 9:57 PM

Private

David•5/7/25, 9:57 PM

Been down for about 2 hours

David•5/7/25, 9:57 PM

Same image as the last 3 days, container is trying to load from runpod cache

DDj You can check on Docker Hub directly to avoid messing with anything. https://ap...

noahpantspartyOP•5/7/25, 9:59 PM

away from kb right now but will check soon.

georg•5/7/25, 10:04 PM

for me, now the whole workers are just gone away and start all over, assigned 40 and they all started over again.

David•5/7/25, 10:17 PM

The issue is to do with the cache

David•5/7/25, 10:17 PM

i'd imagine

David•5/7/25, 10:31 PM

https://uptime.runpod.io/

Reporting no issues for serverless, can we get an update so we can relay to our customers

RunPod status

Welcome to RunPod status page for real-time and historical data on system performance.

zongheng1619•5/7/25, 10:31 PM

yeah, it's tryng to pull the image from cache again and again, I see it's even said success.

Dj•5/7/25, 10:39 PM

We've identified the issue, I'm not sure why we won't fire an incident for it but it's being treated like one internally.

noahpantspartyOP•5/7/25, 10:46 PM

FWIW: I'm still having the same issues. I nuked the endpoint and recreated and still have the same problems.

David•5/7/25, 10:50 PM

This is pretty frustrating, there has been frequent issues with runpod serverless. This 3 hours has cost us once again

Dj•5/7/25, 10:52 PM

I totally understand, the people fixing this issue are actively using this thread to help move things forward. I'll join their call and listen for progress.