Error starting container on serverless endpoint
Hello, I'm having an issue on my serverless endpoint when it starts up.
When our endpoint tries to initialize the container. We get an error saying 'error response from daemon', failed to create task for container and cites an 'out of space' issue. I believe this is coming from Runpod's infra and not something we can resolve.
Can you please advise how we can fix this error, it's causing delays for our customers.
Thanks!
2025-01-04T23:12:32Z Status: Image is up to date for <docker_image>
2025-01-04T23:12:32Z worker is ready
2025-01-04T23:12:33Z start container for <docker_image>: begin
2025-01-04T23:12:33Z error starting container: Error response from daemon: failed to create task for container: failed to create shim task: Could not create the sandbox resource controller mkdir /sys/fs/cgroup/memory/docker/kata_9d38d24b9b1da3f1330e228ba1f6e9e772dba77771c6a074707b710b404a8d27: no space left on device: unknown
2025-01-04T23:12:49Z start container for <docker_image>: begin
2025-01-04T23:12:50Z error starting container: Error response from daemon: failed to create task for container: failed to create shim task: Could not create the sandbox resource controller mkdir /sys/fs/cgroup/memory/docker/kata_9d38d24b9b1da3f1330e228ba1f6e9e772dba77771c6a074707b710b404a8d27: no space left on device: unknown
2025-01-04T23:13:06Z start container for <docker_image>: begin
2025-01-04T23:13:07Z error starting container: Error response from daemon: failed to create task for container: failed to create shim task: Could not create the sandbox resource controller mkdir /sys/fs/cgroup/memory/docker/kata_9d38d24b9b1da3f1330e228ba1f6e9e772dba77771c6a074707b710b404a8d27: no space left on device: unknown
When our endpoint tries to initialize the container. We get an error saying 'error response from daemon', failed to create task for container and cites an 'out of space' issue. I believe this is coming from Runpod's infra and not something we can resolve.
Can you please advise how we can fix this error, it's causing delays for our customers.
Thanks!
2025-01-04T23:12:32Z Status: Image is up to date for <docker_image>
2025-01-04T23:12:32Z worker is ready
2025-01-04T23:12:33Z start container for <docker_image>: begin
2025-01-04T23:12:33Z error starting container: Error response from daemon: failed to create task for container: failed to create shim task: Could not create the sandbox resource controller mkdir /sys/fs/cgroup/memory/docker/kata_9d38d24b9b1da3f1330e228ba1f6e9e772dba77771c6a074707b710b404a8d27: no space left on device: unknown
2025-01-04T23:12:49Z start container for <docker_image>: begin
2025-01-04T23:12:50Z error starting container: Error response from daemon: failed to create task for container: failed to create shim task: Could not create the sandbox resource controller mkdir /sys/fs/cgroup/memory/docker/kata_9d38d24b9b1da3f1330e228ba1f6e9e772dba77771c6a074707b710b404a8d27: no space left on device: unknown
2025-01-04T23:13:06Z start container for <docker_image>: begin
2025-01-04T23:13:07Z error starting container: Error response from daemon: failed to create task for container: failed to create shim task: Could not create the sandbox resource controller mkdir /sys/fs/cgroup/memory/docker/kata_9d38d24b9b1da3f1330e228ba1f6e9e772dba77771c6a074707b710b404a8d27: no space left on device: unknown
