Unstable Serverless GPU performance (Mostly too slow)
Despite using the same specifications and Docker image, the 4090 GPU workers have become unstable.
ComfyUI image-generation tasks used to run at about 4–5 iterations per second, but since 28 June, they’ve been wildly inconsistent. Sometimes 1 iteration per second, and sometimes even 4 seconds per iteration.
I’ve tried terminating the faulty workers and restarting them, even with a fresh Docker image, but the issue persists....
CPU usage rises to 99% even though no processing is being performed
This issue has been observed on the US-IL-1 RTX4090 since around June 27th.
The image used is as follows.
This can be reproduced by starting a serverless worker with the sleep command to prevent it from dying.
runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04...
Bad workers are lurking. Clear differences in processing speed
Despite having the same specs, serverless has workers with obvious problems with processing speed.
All the jobs in the image have the exact same processing, and the exact same DC server specs, but there is a big difference in processing speed.
I terminate these bad workers every day through batch processing, but they keep appearing every day, and there's no end to it.
(US-IL-1 RTX 4090)...
Still waiting
I've opted to deploy my first serverless GPU, I opted for a RTX A5000, and 'bigcode/starcoder'. It's been almost 40 minutes. And I'm still waiting for models to complete. How long does this normal take for a 16B model? My person GPU (4070) was able to load it within seconds.?
