Runpod•2mo ago

Hello my pods can't see the gpu

From yesterday I have an issue with my pod.
First it started with the nvidia-smi command:

root@d6a7d8ca96f6:/# nvidia-smi
Failed to initialize NVML: Unknown Error

root@d6a7d8ca96f6:/# nvidia-smi
Failed to initialize NVML: Unknown Error

root@d6a7d8ca96f6:/# nvidia-smi
Failed to initialize NVML: Unknown Error

root@d6a7d8ca96f6:/# nvidia-smi
Failed to initialize NVML: Unknown Error

I had to restart the pod and everything worked again.
This morning, I add the same issue but wanted to check if I could still do any gpu related processing but nothing worked.

root@d6a7d8ca96f6:/# python
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("torch.cuda.is_available():", torch.cuda.is_available())
torch.cuda.is_available(): False

root@d6a7d8ca96f6:/# python
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("torch.cuda.is_available():", torch.cuda.is_available())
torch.cuda.is_available(): False

root@d6a7d8ca96f6:/# python
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("torch.cuda.is_available():", torch.cuda.is_available())
torch.cuda.is_available(): False

root@d6a7d8ca96f6:/# python
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("torch.cuda.is_available():", torch.cuda.is_available())
torch.cuda.is_available(): False

As you can see on the screen the pod has been up from 2 days ago and it shows that there is a gpu.

I'll use this pod to run my api I can't check all the time if a gpu is available knowing that I'm renting it I guess I shouldn't have any issue related to my gpu being reallocated to someone right ?

Dj•11/7/25, 7:40 PM

This is something that happens sometimes and can be easily resolved by restarting the Pod or deploying a new one. I'll make sure we tend to the GPU in this screenshot though.

mysterion186OP•11/11/25, 2:18 PM

Hi, thanks for the reply, then the solution currently for me is to programmatically restart the pod ? Because the issue happen at random time (happened again this morning, but there were no logs).

Hello my pods can't see the gpu

Similar Threads

Similar Threads

Similar Threads