4xH100 pod is stuck -- can't restart or stop

I am still connected with SSH, but the pod can't be used due to some network issues. RunPod UI also can't reach it (it shows waiting for logs).

Over night the pod failed with:

[E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=9442948, OpType=_ALLGATHER_BASE, NumelIn=196608, NumelOut=786432, Timeout(ms)=1800000) ran for 1800631 milliseconds before timing out.

[E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=9442948, OpType=_ALLGATHER_BASE, NumelIn=196608, NumelOut=786432, Timeout(ms)=1800000) ran for 1800631 milliseconds before timing out.

Wasting a lot of money.

Runpod•2y ago•

5 replies

DreamGen

4xH100 pod is stuck -- can't restart or stop

I am still connected with SSH, but the pod can't be used due to some network issues. RunPod UI also can't reach it (it shows waiting for logs).

Over night the pod failed with:

[E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=9442948, OpType=_ALLGATHER_BASE, NumelIn=196608, NumelOut=786432, Timeout(ms)=1800000) ran for 1800631 milliseconds before timing out.

[E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=9442948, OpType=_ALLGATHER_BASE, NumelIn=196608, NumelOut=786432, Timeout(ms)=1800000) ran for 1800631 milliseconds before timing out.

Wasting a lot of money.

4xH100 pod is stuck -- can't restart or stop

Similar Threads

4xH100 pod is stuck -- can't restart or stop

Similar Threads

Similar Threads

Similar Threads