🆘 We've encountered a serious issue with the machines running in our production environment
🆘 We've encountered a serious issue with the machines running in our production environment on RunPod: the GPU utilization fluctuates wildly, sometimes even dropping to zero, which significantly slows down task execution. Who should I contact?
15 Replies
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
The reason we're using SOS is because we've encountered this issue in a production environment, which directly affects the user experience, but I don't know who to turn to for help.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
During the inference process, we received feedback from users that the inference speed was particularly slow. Upon checking, we confirmed that the issue was indeed related to the inference, but the GPU utilization was either zero or very low.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
Despite all other conditions remaining unchanged, sometimes the inference speed is fast, and at other times it is very slow, even though the model has already been loaded into the GPU memory.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
yeah
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
but gpu kernels are not running at all
inference speed is extremely low
yes
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
GPU utilization fluctuates wildly, sometimes even dropping to zero, and we have nothing changed!
This is going to take up more of our time, and we are short-staffed. I just want to know if Runpod has technical personnel who can help us troubleshoot this issue. We have checked the code logic and found no issues.
how to report to runpod?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
OK
thx
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View