GPU not detected on RunPod serverless - HELP!!
Hey everyone,
I'm running into an issue on RunPod serverless endpoint. Despite having CUDA 12.4.1 set up in my Docker container, my models are initializing on CPU instead of GPU.
My logs show: "Initializing pipeline on cpu"
I'm trying to run Whisper model which would really benefit from GPU acceleration. I've included all the CUDA requirements in my Dockerfile and have torch 2.4.1 in my requirements.txt.
Has anyone encountered a similar issue with RunPod serverless? Is there something specific I need to configure to make it recognize the GPU?
Any help would be greatly appreciated!
Thanks in advance
4 Replies
it greatly depends on your code actually
or the config for your appliction
maybe there's some configuration you need to do for that specific "pipeline"?
maybe check if torch.cuda.is_available() returns true
maybe check if torch.cuda.is_available() returns true
Yeah this returns false and that why it goes to cpu
Try to launch a new pod with cuda 12.4 and later selected
Or else it might be incompatible