Networking Multiple Pods Together
I'm looking to train a distributed model on runpod. When configuring the torch.distributed or jax.distributed you provide a
This consitently fails. My intuition is that the docker containers don't belong to the same network, to my knowledge we users don't have the privilege to setup such a network on the datacenters machine, only modify containers on a one off basis.
Any guidance on enabling communication between pods would be greatly appricieated!
coordinator_address70000ip routepython -m http.server 70000curl <pod_1_local_ip>:<pod_1_70000_port>This consitently fails. My intuition is that the docker containers don't belong to the same network, to my knowledge we users don't have the privilege to setup such a network on the datacenters machine, only modify containers on a one off basis.
Any guidance on enabling communication between pods would be greatly appricieated!
