Guidance on Mitigating Cold Start Delays in Serverless Inference
We are experiencing delays during the cold starts of our serverless server used for inference of a machine learning model (Whisper). The main suspected cause is the download of model weights (custom model trained by us), which are fetched via the Hugging Face package within the Python code. We are exploring possible solutions and need guidance on feasibility and best practices.
Additional Context:
- The inference server currently fetches model weights dynamically from Hugging Face during initialization, leading to delays.
- The serverless platform is being used for inference as part of a production system requiring low latency....
A40 Throttled very regularly!
I have a serverless endpoint with 3GPU that is being fully throttled very regularly.
It is fully unusable for long minutes, see screenshot, request are being queued forever.
It has been the case yesterday and today, it's far too unreliable......
All 16GB VRAM workers are throttled in EU-RO-1
I have a problem in EU-RO-1:
all worked are constantly in throttled state (xz94qta313qvxe, gu1belntnqrflq and so on)...
1s delay between execution done and Finished message
I get almost one second of delay between a console message at the end of my handler and the "Finished" message.
I am wondering why, and how to reduce this....
Serverless is Broken
Something is clearly broken. Delay times are around 2 mins even when the same worker is getting a request in a row, it still takes 2 mins. It's not a cold start issue because even my normal cold starts don't take longer than 15 seconds.
