network volume not persisting, help!
i create a storage in runpod, i hit deploy pod with volume, i pick a 5090 gpu, i pick a comfy ui template, i dont change anything else, i run, everythings good, terminate the pod, all new workflows, all new lors, outputs, gone from the drivve, jupiter doesnt show em at all
what is happening?
28 Replies
ok this has to be the issue, in jupiter, everything is on / not on /workspace
but the default on the pod deployment says /workspace, wtf
running a test.....
hmmm maybe cuz this volume is brand new, it needs there to be a /workspace to begin with? ill try that
like a workspace folder i create manually
nope still the same, wtf
having this issue all week wtf
i literally dont know what to do
@max4c any tech folk around?
@Finley @NERDDISCO any clue?
taking a look
which comfyui template did you use? The runpod official one? Could you share a link to the template?
just type guf in search to find it
@Dj the issue here
It's possible this template is just poorly configured, but it doesn't seem like it would be. Testing now, just waiitng for the Docker Image.
Sorry I misfired on my original answer, I'm still working this out
Okay, I think the template doesn't assume you'll use network storage at all and will just always start this up. It's well made, but it may not be super obvious that it's using/not using Network Storage. When I start a Pod with this template, I'm placed into
/app/ComfyUI, not /workspace.so how do i get it to store on a network storage?
i guess i have to look for templates that are labeled for network storage?
The template developer is polite enough to have left an email, I can drop them a message if you want, and in the meantime you can
cp any file you want from the ComfyUI directory to the /workspace
Some of them say this, some will say "PERMANENT". One of our employees makes his own ComfyUI template unless you specifically wanted wan2.2 preloaded for you
https://console.runpod.io/hub/template/better-comfyui-cuda12-full?id=ctuuw1qab8
This is @Madiator2011 (Work)'sill try it, and yes i just want a consistent space at all i can build on
if i accidently load another template, will my old volume files be deleted?
well another question
ill try this right now
@Dj so i cant use that guf workspace permanently really right? the one i originally was trying to use. if i move all files into a folder that says /workspace, when i go to run that template, itll install all its stuff on / ,again right?
Yes :frowning3: but you'll have a copy in your workspace.
ok
gonna try this one : "ComfyUI Manager Permanent Storage"
well i guess doesnt matter
they really should tell you some templates dont let you use network storage, thats wild that they dont
when i lost my entire first comfy setup, i was devestated, all outputs and workflows i created, gone forever
20gb, whats in his setup jesus christ
wow thats just the docker itself
@Dj is he using an old cuda by chance, i think wan might be only 12.8+?
My team is working on redesigning what templates look like on Runpod I think this would be super important. @max4c
The CUDA version shouldn't matter for storage, for CUDA if Comfy can start everything's fine.
i guess this is how you spot if they use perm storage or not?
this template should check all the bells and whistles https://console.runpod.io/hub/template/wan2-2-with-5b-model-preinstalled-comfyui?id=90xkj1p4ce
"Fresh Install
The base template is Pytorch 2.4. Should be stable for most machines (CUDA version)
ComfyUI and ComfyUI Manager are installed in the permanent disk"
i really hope this works
spent all week on this ðŸ˜
oh my god i see /workspace finally
i thought it was just a myth
killing pod
rerunning
holy f that loaded up fast
servers really chugging now
workflow savedddd
finalllllllyyyyyyyyyyyy
:RainbowWeeb: :RainbowWeeb: :RainbowWeeb: :RainbowWeeb: :RainbowWeeb: :RainbowWeeb: :RainbowWeeb: :RainbowWeeb:
loading the comfy interface really choking tho
@Dj is runpod having server issues?
The proxies can be a little weird, but not to my knowledge
We're doing a little maintenance on each DC but it's nothing a user would notice else we'd post to #🚨｜incidents
ill try on a different browser
nope still chugging
maybe i should try on a new pod
yep just that pod i guess
loaded faster now
upload speeds to jupiter are rough
i have to upload a 20gb diffusion model, no other faster way than jupiter? @Dj
the upload speeds are super slow
I would recommend using SFTP if you could
Using Jupyter's dashboard sends it through a proxy and yeah, its rough
please tell me how i tried setting up runpod ctl and its crap
documentation not working on windows
rsync, which requires WSL
And then there's this tool made by one of our engineers:
https://docs.runpod.io/community-solutions/ohmyrunpod/overview
stuff like this is why ai will be extremely gatekept
We're working on that :) Just takes time
traceback error
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\dfgdfgdfgdf\AppData\Local\Programs\Python\Python313\Scripts\OhMyRunPod.exe\__main__.py", line 2, in <module>
from OhMyRunPod.main import main
File "C:\Users\dfgdfgdfgdfg\AppData\Local\Programs\Python\Python313\Lib\site-packages\OhMyRunPod\main.py", line 6, in <module>
from OhMyRunPod.modules.file_transfer import show_file_transfer_menu
File "C:\Users\dfgdfgdgf\AppData\Local\Programs\Python\Python313\Lib\site-packages\OhMyRunPod\modules\file_transfer\__init__.py", line 1, in <module>
from .file_transfer import setup_croc, setup_sftp, show_file_transfer_menu
File "C:\Users\dfgdfgdfgdf\AppData\Local\Programs\Python\Python313\Lib\site-packages\OhMyRunPod\modules\file_transfer\file_transfer.py", line 8, in <module>
import termios
ModuleNotFoundError: No module named 'termios'
PS C:\Users\gdfgdfgdgdfgdf>
replaced user with dsfnkjsdnf
i did the install on windows terminal
all while runpod is charging me money to navigate and troubleshoot their own software
they gotta give people refunds or discounts or something at some point for all this
@Dj
im just gonna try installing through comfy
altho i guess thats still a proxy right so
probably still really bad
solution: use civiccomfy node from comfyui manager
downloaded in settings, no shiity terminal command line crap
a nice lovely gui thatll tell you it loves you after it works successfullyOhMyRunPod is not made to run on your PC 🙂
@Atlas did it work out in the end? we currently "only" make sure that the official template are working with all the things we do provide like network storage, all of the community stuff is "out of our control", so it can happen that those are not using the features that exist in the platform. but i totally get that this is something that you as a user doesn't want to deal with and we are working very hard to get this stuff solved, so that you can just use everything without debugging it yourself
Ummmm i got it working. Put a warning about not all templates have permanent storage designed in (NOT MENTIONED IN 17 LAYERS OF WIKI HELP PAGES) i posted a screenshot above about how you can tell with the different storage logos. The notice should be right next to the template button when you go to deploy or in that template search menu.
Everything else seems to be working, the transfer speeds is extremely bad for Jupiter, i found custom plugins in comfy to solve that but having to whip out 10 different command line codes just to transfer a file bigger than 50mb is crazy
No cli stuff
If yall want people to stop losing their minds and leaving, gotta make the ui. Everytime I look at a windows power shell terminal, i ponder All the mistakes I've made in my life that somehow led me to this point
User friendly= $$$$$$$$$$$$
Huge priority should be being able to access your permanent network storage without having to launch a pod or a terminal, simple drag and drops. Having to pay money And hog a GPU to access It is crazy
Because I am not navigating using terminal, I don't have the time
There should be a button that says access storage right next to deploy with pod
Also get some more a40s in the usa servers
totally agree with you on all points