H100 PCIe pods: NVIDIA driver/CUDA init issue (Torch can’t see GPU, H100 NVL OK)
Hey Runpod team, I’m seeing what looks like a driver/CUDA setup issue only on H100 PCIe pods (not H100 NVL).
Setup
* Pod GPU: 1× H100 PCIe
* Official template:
*
*
*
Problem
Even though
Output on H100 PCIe:
So
I tried multiple times with a different pod each time.
Control / comparison (works)
Same exact template, but GPU: 1× H100 NVL
*
*
* Same script output:
Conclusion
This seems specific to the H100 PCIe host driver/CUDA setup (570.195.03 / CUDA 12.8) or how the container is wired to the host on those nodes. NVL nodes (550.107.02 / CUDA 12.4) work fine with the same image.
Happy to provide a pod ID / logs if you tell me what you need.
Setup
* Pod GPU: 1× H100 PCIe
* Official template:
runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404*
pip freezetorch==2.8.0+cu128*
nvidia-smi*
Driver Version: 570.195.03 CUDA Version: 12.8Problem
Even though
nvidia-smiOutput on H100 PCIe:
So
torch.cuda.is_available()device_countI tried multiple times with a different pod each time.
Control / comparison (works)
Same exact template, but GPU: 1× H100 NVL
*
nvidia-smi*
Driver Version: 550.107.02 CUDA Version: 12.4* Same script output:
Conclusion
This seems specific to the H100 PCIe host driver/CUDA setup (570.195.03 / CUDA 12.8) or how the container is wired to the host on those nodes. NVL nodes (550.107.02 / CUDA 12.4) work fine with the same image.
Happy to provide a pod ID / logs if you tell me what you need.
