How to transfer between pods?
I'm running stable diffusion and would like to transfer my outputs to a different pod to continue working. When using runpodctl to transfer data between from 1 pod to another, what is the command? I have tried using runppdctl send “file path name” but this isn’t working for me. What file path should I be using? Can someone share an example of the file paths structure, please? It was suggested I post the question here, I'm not getting an error, it's just that nothing is happening.
18 Replies
https://discord.com/channels/912829806415085598/948767517332107274/1194777851896016906
Just relinking here to continue the conversation here
and then also, if that doesn't work this is just images right?
You could try to:
SSH into the pod and then SCP the files locally to your computer. And then always SCP it back to another pod (you want to zip the folder to make it easier to transfer over a bunch of stuff, also copying over one file is more efficient than many files):
https://discord.com/channels/912829806415085598/1194711850223415348
https://chat.openai.com/share/5f3ca32a-9c32-4ca2-a87e-bf1039bf44f3
ChatGPT
A conversational AI system that listens, learns, and challenges
But yeah, my guess is maybe.. runpodctl --help, to verfy the cli is working.
runpodctl send ./output since you are alread at the path and output directory is right thereThank you! I'll check out the chat you shared.
Is anyone else from the mod team or runpod able to weigh in on this. Runpodctl is marketed as being able to transfer data from one pod to the next, but I can’t easily find the command line syntax to use in Jupyter lab terminal to do this. I’m easily able to do it from my laptop to the pod and back. I just need help with pod to pod transfer. May I request that this be clearly marked in the documentation, as well. There are often issues with pods going down or being unavailable after they are stopped. Thank you!
@flash-singh / @Justin maybe they will know why runpodctl send isnt displaying anything? That is weird
My other theory @MonetStarr Is you do send without a “” around the paths? lol. I dont know if that will make a difference just seeing the docs their send doesnt have a quote
did it work without the ""?
Hi no, it didn’t work without the quotes either. 😩
I tried without quotes too. That’s why I was thinking I’m not using the correct file path structure.
Hm. What do you have in there again?
is it just images?
How big is this folder?
Im thinking if it is small enough, maybe the objective is just secure your stuff first than get runpodctl to work
A small tip is that you can usually do runpodctl send [first letter of your folder] and tap [TAB] button to autocomplete and see if that works
I do see it working for me
i just gave it a try by creating a test.txt file, and then also creating a directory, putting a test.txt file inside, and trying to send the directory over
My immediate thoughts are:
1) Can you zip and download it manually through jupyter server if small enough it should be good
https://docs.runpod.io/docs/download-a-folder-from-a-pod
2) Can you scp it
https://docs.runpod.io/docs/transfer-files-with-rsync
3) Can you push it to git repository
You could try to open up a web ssh terminal and see if that makes a difference 🤷, prob wont but just a guess
Yes, just images. Maybe a couple hundred. I already terminated the pod, but I’ve run into this a couple times now and would like to get sorted for future reference.
Thank you! I will look into zipping and downloading manually next time I have a pod running.
Hm. I do find it strange that it is happening to you as it doesn't happen to me... HM. But I do recommend, maybe considering, using something like a network storage, if you are more programatic, I like to use Firebase File Storage and I manually run a code to upload to an API server or something.
Hopefully u can get SCP working!
https://discord.com/channels/912829806415085598/1194711850223415348/1194711850223415348
I do think personally, this is also just a great way b/c you cut out any middleman
it's just as direct to the computer as you can
That’s my challenge I’m not very programmatically inclined 😂
I follow tutorials and directions really well though. ☺️
I see! Maybe try the Easy way to ssh to a Pod, that will establish a private/public key pairing for you in the future 🙂 and then in the future should be able to just, follow along with this:
https://docs.runpod.io/docs/transfer-files-with-rsync
The public key / private key will automatically be stored wherever the tutorial is telling you
RunPod
Transfer Files with SCP/rsync
Prerequisites/SetupNote: your local machine must be running Linux or a WSL instance in order to use rsync. Make sure your Pod is configured to use real SSH. Refer to this guide for assistance.If you intend to use rsync, make sure it's installed on both your local machine and your Pod with apt instal...
such as this
And can have chatgpt fill in the rest is what I did lol
Im not a genius at the SCP thing either / understand the whole key pairing thing. all ik is that once i did the easy ssh stuff, my keys are wherever the pod told me in that picture, and i could do stuff
def something to play with 🙂
The only other thought I have is in the future, zipping your folder, and pushing it to github lul.
Since it just images
Thank you !
send needs local file , use
/workspace/..
workerspace isnt a folder in outputThank you!