Sulove
RRunPod
•Created by Sulove on 4/4/2025 in #⚡｜serverless
GPU not detected on RunPod serverless - HELP!!
Hey everyone,
I'm running into an issue on RunPod serverless endpoint. Despite having CUDA 12.4.1 set up in my Docker container, my models are initializing on CPU instead of GPU.
My logs show: "Initializing pipeline on cpu"
I'm trying to run Whisper model which would really benefit from GPU acceleration. I've included all the CUDA requirements in my Dockerfile and have torch 2.4.1 in my requirements.txt.
Has anyone encountered a similar issue with RunPod serverless? Is there something specific I need to configure to make it recognize the GPU?
Any help would be greatly appreciated!
Thanks in advance
9 replies
RRunPod
•Created by Sulove on 11/23/2024 in #⚡｜serverless
RunPod GPU Availability: Volume and Serverless Endpoint Compatibility
Hey everyone! Quick question about RunPod's GPU availability across different deployment types. I'm a bit confused about something:
I created a volume in a data center where only a few GPU types were available. But when I'm setting up a serverless endpoint, I see I can select configs with up to 8 GPUs - including some that weren't available when I created my volume.
Also noticed that GPU availability keeps fluctuating - sometimes showing low availability and sometimes none at all. So I'm wondering:
1. What happens if I pick a GPU type for my serverless endpoint that wasn't originally available in my volume's data center?
2. If I stick to only the GPUs that were available when creating my network volume, how does that work when those GPUs suddenly show low/no availability?
Just trying to understand how RunPod handles these scenarios. Would really appreciate any insights! 🤔
Thanks in advance!"
3 replies