CUDA not recognized

Niels_ · 2023-12-29T15:18:06.441Z

Hi, I'dl like to run a PyTorch model on a GPU, but for some reason the GPU is not recognized by PyTorch even though I'm specifying a template that includes CUDA.

N

Niels_OP•12/29/23, 3:18 PM

I'm getting the following:

>>> import torch
>>> torch.cuda.is_available()
/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0

>>> import torch
>>> torch.cuda.is_available()
/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0

>>> import torch
>>> torch.cuda.is_available()
/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0

>>> import torch
>>> torch.cuda.is_available()
/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0

`

N

Niels_OP•12/29/23, 3:18 PM

The template I'm using is runpod/pytorch:2.1.1-py3.10-cuda12.1.1-devel-ubuntu22.04

J

Justin Merrell•12/29/23, 3:31 PM

May you run nvidia-smi in the console?

N

Niels_OP•12/29/23, 4:23 PM

J

Justin Merrell•12/29/23, 4:51 PM

What is the pod id?

N

Niels_OP•12/29/23, 5:09 PM

fho00ms04wxzxw

S

sane•12/30/23, 11:07 PM

I have the same problem with NVIDIA 4090 pods and cuda 12.1 / torch 2.1

S

sane•12/30/23, 11:10 PM

pods with other GPUs and same cuda / torch versions works well

CUDA not recognized

Similar Threads

Similar Threads

Similar Threads