R
Runpod•6mo ago
bghira

torch cuda shows no devices available (B200)

on 8x B200 system lz1ew4cgoiot8f : 
Python 3.11.11 (main, Dec  4 2024, 08:55:07) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> torhc.__
KeyboardInterrupt
>>> torch.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'torch' is not defined
>>> import torch
>>> torch.__version__
'2.8.0.dev20250319+cu128'
>>> torch.cuda.is_available()
/usr/local/lib/python3.11/dist-packages/torch/cuda/__init__.py:174: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
False
>>> 
Python 3.11.11 (main, Dec  4 2024, 08:55:07) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> torhc.__
KeyboardInterrupt
>>> torch.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'torch' is not defined
>>> import torch
>>> torch.__version__
'2.8.0.dev20250319+cu128'
>>> torch.cuda.is_available()
/usr/local/lib/python3.11/dist-packages/torch/cuda/__init__.py:174: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
False
>>>
6 Replies
bghira
bghiraOP•6mo ago
and nvidia smi shows

message.txt

Poddy
Poddy•6mo ago
@bghira
Escalated To Zendesk
The thread has been escalated to Zendesk!
bghira
bghiraOP•6mo ago
maybe it was an issue with the image? someone on my team found a workaround
Dj
Dj•6mo ago
I'm glad you got it, you didn't make a ticket so we're good.
sirouk
sirouk•5mo ago
what was the workaround for this?
EvoBits
EvoBits•5mo ago
There is an Nvidia driver bug with B200 and kernels newer than 5.15 . Not sure if this was the case here, but we have seen similar issues when running on a kernel newer than 5.15 . Kernel 5.15.* works.

Did you find this page helpful?