R
RunPod•6mo ago
echozhou

How to use runpod for multi-machine distributed training?

We have applied symmetric ports for multiple machines and configured them to be accessible to each other via ssh. But it is not possible to do distributed training. The commands we are using : torchrun --nproc_per_node=1 \ --nnodes=2 \ --node_rank=0 \ ---master_addr="216.249.100.66" \ --master_port=12619 \ test.py torchrun --nproc_per_node=1 \ --nnodes=2 \ --node_rank=1 \ ---master_addr="216.249.100.66" \ --master_port=12619 \ test.py We are using RunPod Pytorch 2.1.
2 Replies
ashleyk
ashleyk•6mo ago
I assume the IP and port are the public IP and public port mapping for SSH?
Levi Morin
Levi Morin•4mo ago
Hi @echozhou , @ashleyk is it possible to do distributed training across multiple runpod machines?
Want results from more Discord servers?
Add your server
More Posts
Check if a pod is idleHello, I am an admin for a team looking to start using runpod. I would like to be able to check whetthe bloke and llm not workingI am brand new to runpod. I just opened an account yesterday and put $25 on it. I used an 80g A100 aLarge discrepancy in broadband available and broadband usedI launched a pod which claims to have about 1688 Mbps download speed. However, when loading a model reproducible: pods crash 50% of the timei am trying to build an API which allows people without big GPUs to run googles weather forcasting mImport PIL (pillow image library) in rp_handler.pyHey everyone, i try to use a tool from the PIL module to process my generated image before sending bserver problemhello, since a few hours the pod: bhhr17bf8l967t has network errors. The mail stated i should contacPossible error in docs: Status of a job with python codeIn the docs, there is this command to retrieve the status of a submitted job: ``` curl https://api.rSlow Download Speed as well - over 6 hours of downloading 4+ GB of files (and still running.).I did a very large batch of images last night and have been trying to download them for the past hou