sh: 1: accelerate: not found

today im getting this error when using Kohya to caption images or trying to train a LoRA using a Stable Diffusion Kohya_ss ComfyUI Ultimate Pod. I have been using these pods for the last month without issue, but today nothing is working. My work flow has not change. I was successfully creating 2 days ago. Any ideas?
Thanks sh: 1: accelerate: not found
and /bin/sh: 1: accelerate: not found
23 Replies
justin
justin5mo ago
Do u have a pod id? Maybe share it. Also I assume this replicates even when creating a new pod? I can’t say as Im just another member, but hopefully when tmmrw PST wraps around u can ping a staff and they can look at ur pod id and logs
NonsenceMan
NonsenceMan5mo ago
Ive tried 3 pods, and each time it takes about 30mins for the new pod to be ready.. I just closed my last pod 20secs ago same on all pods, 3 different setups/builds Yeah, I was thinking I must be in a different time zone and everyone is alseep
justin
justin5mo ago
Sg! Whether u have a pod id, or just share the template link 🔗 i think is good :). Or steps u did to replicate Like: Commands u ran Will be helpful to flag this to them - otherwise they cant really say. Or is this just completely happening in the logs and stopping it from ever starting up? If the latter prob just can share the template name Ah this looks like ashelyks pod https://hub.docker.com/r/ashleykza/stable-diffusion-webui/tags
NonsenceMan
NonsenceMan5mo ago
the Template is Stable Diffusion Kohya_ss ComfyUI Ultimate its happenig while using Kohya_SS to either captions images or train a LoRA. it fails to start the process and just ends.
My workflow and set up hasnt changed and i was succesfull 2 days ago on the same type of Pods.
justin
justin5mo ago
Why dont u default back down to 3.11.0 I think check the template for the docker image + tag and deploy ur own to unblock for now @ashleyk - just fyi - maybe something wrong with ur 3.11.1 It makes sense cause looks like there is a minor version bump 9 hours ago, maybe caused an issue
NonsenceMan
NonsenceMan5mo ago
Im not very tech savvy that way.. I figgured most stuff out by following youtube videos. This is all new to me this month.
justin
justin5mo ago
I see.
No description
justin
justin5mo ago
hopefully u can see this name? U can make ur own custom template with the same settings before u deploy do edit settings or something screenshot it then u can make ur own custom template and in the docker image place just put the same thing but do name…:3.11.0
justin
justin5mo ago
9 hours ago there was a push which may be causing ur issue
No description
justin
justin5mo ago
Here in the container image just copy the name into it and u can set the storage space / ports / all that to the screenshot u take before u deploy the original template
No description
NonsenceMan
NonsenceMan5mo ago
Like this?
No description
NonsenceMan
NonsenceMan5mo ago
Mad ea new template, Im starting the pod up now.. Will see how it goes I think you 100% correct about the Push update 9 hrs ago. Thanks for you help If your still around.. the Pod is up but when trying to log into Jupyter, the password isnt set up its asking for a Token How do i find this?
justin
justin5mo ago
No description
justin
justin5mo ago
GitHub
GitHub - ashleykleynhans/stable-diffusion-docker: Docker image for ...
Docker image for Stable Diffusion WebUI with ControlNet, After Detailer, Dreambooth, Deforum and roop extensions, as well as Kohya_ss and ComfyUI - GitHub - ashleykleynhans/stable-diffusion-docker:...
justin
justin5mo ago
Can try this! The jup1ter thingy is prob the password
NonsenceMan
NonsenceMan5mo ago
thats what it normaly is but isnt working for this template -apprently Jup1t3R!
justin
justin5mo ago
U can try: to open a webssh terminal and do jupyter server list and see if u get a token through there
justin
justin5mo ago
GitHub
GitHub - justinwlin/FooocusRunpod
Contribute to justinwlin/FooocusRunpod development by creating an account on GitHub.
justin
justin5mo ago
ex on mine
justin
justin5mo ago
No description
NonsenceMan
NonsenceMan5mo ago
okay. everything is working, so far all good.. Thanks
justin
justin5mo ago
Glad was able to figure out a work around! And from now on u can guarantee at least a stable version 🙂 if u want to get more technical 😂 u can download docker, pull down ashelyk’s image, retag it to ur username/imagename:tag, and push to ur own repo can ask chatgpt for details ull guarantee always having a copy of the image for ur own stability but for now this also good haha It def is bit daunting at first maybe another’s day adventure lol
ashleyk
ashleyk5mo ago
Apologies, tried to fix an issue with syncing to network volumes and I hard-coded the fix_venv.sh script to fix venvs residing in /workspace/venv by accident and this caused issues with Kohya_ss because its venv is in /workspace/kohya_ss/venv and not /workspace/venv. Its been fixed in 3.11.2.