Runpod•13mo ago

Error response from daemon: container

After uploading my ED25519 SSH, creating a pod (using the "winglian/axolotl-cloud:main-latest image"), and trying to SSH into it, I immediately get a :

Error response from daemon: container [..id..] is not running

Error response from daemon: container [..id..] is not running

Error response from daemon: container [..id..] is not running

Error response from daemon: container [..id..] is not running

error after successfully authenticating to ssh.runpod.io with the public key. Looking at the pod's system logs, it says the image was created, but it doesn't get beyond starting it and the container log shows a "curl: no URL specified" error:

Container logs:
2024-12-14T23:05:26.009017152Z Exporting environment variables...
2024-12-14T23:05:26.033371400Z  * Starting OpenBSD Secure Shell server sshd
2024-12-14T23:05:26.043064825Z    ...done.
2024-12-14T23:05:26.051583850Z curl: no URL specified!
2024-12-14T23:05:26.051609264Z curl: try 'curl --help' or 'curl --manual' for more information
[REPEATED]

Container logs:
2024-12-14T23:05:26.009017152Z Exporting environment variables...
2024-12-14T23:05:26.033371400Z  * Starting OpenBSD Secure Shell server sshd
2024-12-14T23:05:26.043064825Z    ...done.
2024-12-14T23:05:26.051583850Z curl: no URL specified!
2024-12-14T23:05:26.051609264Z curl: try 'curl --help' or 'curl --manual' for more information
[REPEATED]

Container logs:
2024-12-14T23:05:26.009017152Z Exporting environment variables...
2024-12-14T23:05:26.033371400Z  * Starting OpenBSD Secure Shell server sshd
2024-12-14T23:05:26.043064825Z    ...done.
2024-12-14T23:05:26.051583850Z curl: no URL specified!
2024-12-14T23:05:26.051609264Z curl: try 'curl --help' or 'curl --manual' for more information
[REPEATED]

Container logs:
2024-12-14T23:05:26.009017152Z Exporting environment variables...
2024-12-14T23:05:26.033371400Z  * Starting OpenBSD Secure Shell server sshd
2024-12-14T23:05:26.043064825Z    ...done.
2024-12-14T23:05:26.051583850Z curl: no URL specified!
2024-12-14T23:05:26.051609264Z curl: try 'curl --help' or 'curl --manual' for more information
[REPEATED]

I'm trying to run an LLM finetune using axolotl

Solution

Quick update, this is due to the template in the tutorial no longer being maintained as the repo for git moved under axolotl.
The working RunPod Template for this is axolotali/axolotl-cloud:main-latest
And currently the axolotlai template is working. I have raised this issue internally with our team and we will get the template in the tutorial updated to point to this as well.

Jump to solution

Jason•12/18/24, 12:50 AM

Eh I kind kf don't get it, is the error response from daemon container or this curl command

Jason•12/18/24, 12:51 AM

If it's just the curl command then it's because the curl command arguments isn't valid

Jason•12/18/24, 12:51 AM

The (--paramname ) thing

!XabbuOP•12/18/24, 2:41 AM

This error is in response from me starting the pod and from the curl command executed from within the starting pod. All I did was start the pod. Is that not supposed to work straight out of the box?

Jason•12/18/24, 2:46 AM

What template did you use?

Jason•12/18/24, 2:46 AM

Does your docker image works with that command?

!XabbuOP•12/18/24, 2:49 AM

I used LazyAxolotl (winglian/axolotl-cloud:main-latest)

!XabbuOP•12/18/24, 2:50 AM

per Fine tune an LLM with Axolotl on RunPod

Fine tune an LLM with Axolotl on RunPod | RunPod Documentation

Learn how to fine-tune large language models with Axolotl on RunPod, a streamlined workflow for configuring and training AI models with GPU resources, and explore examples for LLaMA2, Gemma, LLaMA3, and Jamba.

!XabbuOP•12/18/24, 2:50 AM

All I'm trying to do is launch an axolotl run once I can get into the server, send my files over, etc.

!!Xabbu After uploading my ED25519 SSH, creating a pod (using the "winglian/axolotl-clou...

Jason•12/18/24, 2:51 AM

I see, did you set any environment variable or anything that's need to be configured here it seems like it says no specified url

Jason•12/18/24, 2:52 AM

I can try it like tommorow to follow that and check if you want or you wanna ask staff in a ticket

!!Xabbu After uploading my ED25519 SSH, creating a pod (using the "winglian/axolotl-clou...

PoddyAPP•12/18/24, 2:52 AM

@!Xabbu

Escalated To Zendesk

The thread has been escalated to Zendesk!

!XabbuOP•12/18/24, 2:53 AM

I just created a volume in a region, picked a GPU, selected # of GPUs, and started the pod with that image/template

JJason I can try it like tommorow to follow that and check if you want or you wanna as...

!XabbuOP•12/18/24, 2:53 AM

I would appreciate that

Titanium-Monkey•12/18/24, 9:13 PM

curl command is coming from Container Start Command in this Template, its trying to call an ENV VAR called $SCRIPT which does not exist on this image while running it locally.
I would expect to see an env var from the command printenv | grep SCRIPT but there is nothing. Not a single mention of a URL in env.
Remove the Container start command by Editing the Pod and the container should start and be accessible, as it stops the container from continually looping. You can also keep a container from looping by adding bash -c ‘sleep infinity’ to the container start command.
As once the command finishes, it will just restart the pod unless its told to stay up.
https://github.com/axolotl-ai-cloud/axolotl

Titanium-Monkey•12/18/24, 9:13 PM

Solution

Titanium-Monkey•12/18/24, 11:23 PM

!XabbuOP•12/18/24, 11:28 PM

Excellent. Thanks. I will give the other template a try.

!XabbuOP•12/19/24, 6:24 PM

I was able to stand up the pod with this change. Thanks

Error response from daemon: container

Similar Threads

Similar Threads

Similar Threads