Container images loading stuck in a loop when launching
I have workers that never actually launch after pulling containers. Have no idea how to debug this. Deleted and recreated the endpoint and get the same behavior. Any thoughts on how to resolve? It is extra aggrevating as I've had to spin this up because of the EU-SE-1 performance degradation and now getting hit with this issue.
endpoint id: d9b5s5qpbl0sfb
=== Snip from the logs. This is just looping repeatedly. Have the credentials set for dockerhub, the image is published and available, etc. ===
2025-05-07T19:34:02Z loading container image from cache
2025-05-07T19:34:50Z Loaded image: docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z 0.1.24-dev0 Pulling from docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z Digest: sha256:b066e7235b92701dca45b26a3da6437e1fdc3ca96f751fd5bd614cdb40f532bb
2025-05-07T19:34:51Z Status: Image is up to date for docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z worker is ready
2025-05-07T19:37:23Z create container docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:23Z loading container image from cache
2025-05-07T19:37:31Z create container: still fetching image docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:32Z create container docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:32Z create container: still fetching image docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:33Z Loaded image: docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z 0.1.24-dev0 Pulling from docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z Digest: sha256:b066e7235b92701dca45b26a3da6437e1fdc3ca96f751fd5bd614cdb40f532bb
2025-05-07T19:37:34Z Status: Image is up to date for docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z worker is ready
endpoint id: d9b5s5qpbl0sfb
=== Snip from the logs. This is just looping repeatedly. Have the credentials set for dockerhub, the image is published and available, etc. ===
2025-05-07T19:34:02Z loading container image from cache
2025-05-07T19:34:50Z Loaded image: docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z 0.1.24-dev0 Pulling from docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z Digest: sha256:b066e7235b92701dca45b26a3da6437e1fdc3ca96f751fd5bd614cdb40f532bb
2025-05-07T19:34:51Z Status: Image is up to date for docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:34:51Z worker is ready
2025-05-07T19:37:23Z create container docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:23Z loading container image from cache
2025-05-07T19:37:31Z create container: still fetching image docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:32Z create container docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:32Z create container: still fetching image docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:33Z Loaded image: docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z 0.1.24-dev0 Pulling from docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z Digest: sha256:b066e7235b92701dca45b26a3da6437e1fdc3ca96f751fd5bd614cdb40f532bb
2025-05-07T19:37:34Z Status: Image is up to date for docker.io/REDACTED_NAMESPACE/REDACTED_IMAGE:REDACTED_TAG
2025-05-07T19:37:34Z worker is ready
