GPU not detected on RunPod serverless - HELP!!
Hey everyone,
I'm running into an issue on RunPod serverless endpoint. Despite having CUDA 12.4.1 set up in my Docker container, my models are initializing on CPU instead of GPU.
My logs show: "Initializing pipeline on cpu"
I'm trying to run Whisper model which would really benefit from GPU acceleration. I've included all the CUDA requirements in my Dockerfile and have torch 2.4.1 in my requirements.txt.
Has anyone encountered a similar issue with RunPod serverless? Is there something specific I need to configure to make it recognize the GPU?
Any help would be greatly appreciated!
Thanks in advance
4 Replies
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
maybe check if torch.cuda.is_available() returns true
maybe check if torch.cuda.is_available() returns true
Yeah this returns false and that why it goes to cpu
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View