Requests Stuck in Queue After Docker Image Update
When I updated my GitHub repository (which is used in my serverless setup to build/deploy my Docker image), users should've still been able to generate images using the existing Docker image during the update process. However, while the Docker image was being updated, image generation through my RunPod serverless setup became completely unavailable. Requests were sent to RunPod but remained in the queue for an extended period, and I eventually had to cancel them to avoid incurring unnecessary costs.
To resolve this, I terminated the current workers so that new ones would launch with the updated Docker image. This part worked as expected, and new workers were created using the latest Docker image. However, the issue persisted unfortunately. Image generation requests continued to get stuck in the queue, and I was still unable to generate any images.
What made this especially frustrating was that I could not find any single log about it, so I had no way to troubleshoot the issue myself. I would love to see more logs on serverless to troubleshoot such issues if they happen again.
Strangely, when I checked the project again a day later, everything was functioning correctly. I still do not know what caused the failure after all workers were updated and running the new Docker image. The only change I made in the repository was the addition of more input images, which were included in the Docker build.
To resolve this, I terminated the current workers so that new ones would launch with the updated Docker image. This part worked as expected, and new workers were created using the latest Docker image. However, the issue persisted unfortunately. Image generation requests continued to get stuck in the queue, and I was still unable to generate any images.
What made this especially frustrating was that I could not find any single log about it, so I had no way to troubleshoot the issue myself. I would love to see more logs on serverless to troubleshoot such issues if they happen again.
Strangely, when I checked the project again a day later, everything was functioning correctly. I still do not know what caused the failure after all workers were updated and running the new Docker image. The only change I made in the repository was the addition of more input images, which were included in the Docker build.
