How to Speed Up S3 Upload or Make it Async in RunPod Serverless Deployments
I am currently exploring using RunPod as our primary in-house model deployment platform instead of Replicate (our current preferred platform). Our in-house models mostly are txt2img/img2img custom models.
One of the issues I'm facing while testing RunPod is long S3 upload times. For example, for one of our processes, the prediction time is ~1 second, but the S3 upload is taking up to 4-5 seconds (depending on image size), significantly increasing the overall prediction time.
This causes two main problems:
- Long prediction times despite the GPU being free after just 1 second of actual processing
- Increased queue times as workers remain occupied during these long uploads
Is there a way to speed up S3 uploads? Is there a way to make the S3 upload async so that the server can handle multiple concurrent requests? For comparison, Replicate provides temporary File URLs (persistent for ~30 mins) that avoid the S3 upload overhead, resulting in much faster overall request times.
Note: I am already using RunPod's specific S3 upload function that uses chunking.
One of the issues I'm facing while testing RunPod is long S3 upload times. For example, for one of our processes, the prediction time is ~1 second, but the S3 upload is taking up to 4-5 seconds (depending on image size), significantly increasing the overall prediction time.
This causes two main problems:
- Long prediction times despite the GPU being free after just 1 second of actual processing
- Increased queue times as workers remain occupied during these long uploads
Is there a way to speed up S3 uploads? Is there a way to make the S3 upload async so that the server can handle multiple concurrent requests? For comparison, Replicate provides temporary File URLs (persistent for ~30 mins) that avoid the S3 upload overhead, resulting in much faster overall request times.
Note: I am already using RunPod's specific S3 upload function that uses chunking.
