vLLM Inconsistently Hangs at NCCL Initialization
Hi, I am trying to run vLLM on 2x A40s GPUs and it will sometimes hang at NCCL initialization. This inconsistently occurs and sometimes will work fine. But for a pod that it hangs on, repeated attempts will aways hang...
CUDA 12.4.1
python 3.10
vllm 0.7.3
command:
CUDA 12.4.1
python 3.10
vllm 0.7.3
command:
vllm serve unsloth/Meta-Llama-3.1-8B --tensor-parallel-size 2
