Guidance on Mitigating Cold Start Delays in Serverless Inference

We are experiencing delays during the cold starts of our serverless server used for inference of a machine learning model (Whisper). The main suspected cause is the download of model weights (custom model trained by us), which are fetched via the Hugging Face package within the Python code. We are exploring possible solutions and need guidance on feasibility and best practices. Additional Context: - The inference server currently fetches model weights dynamically from Hugging Face during initialization, leading to delays. - The serverless platform is being used for inference as part of a production system requiring low latency. - We offer streaming inference, where low latency is critical for usability. Currently, many calls experience delays exceeding 5 seconds, making the solution unfeasible for our purposes (check img showing the delay times from this month). Solution Options 1. Injecting the Model Directly into the Docker Container This involves embedding the fully trained model within the Docker container. The server would bring up the container with the model file already included. Cons: This will result in significantly larger Docker images. Questions: - Will this approach impact cold-start times, considering the increased image size? - Are there recommended limits on Docker image sizes for serverless environments? 2. Using Storage - Disk Volume Store the model weights on a disk volume provided by the cloud provider. The serverless instance would mount the storage to access the weights. Cons: Potential additional storage costs. Questions: - Does the serverless platform support disk volume storage? We could only find documentation about using storage with Pods. - If supported, is mounting disk volume storage expected to improve cold-start performance? 3. Using Storage - Network Storage Host the model weights on an intranet storage solution to enable faster downloads (compared to public repositories like Hugging Face). Cons: Possible network storage costs and additional management overhead. Questions: - Does the serverless platform support network storage for serverless instances? Again, documentation appears focused on Pods. - Are there recommendations or best practices for integrating network storage with serverless instances? We would like some guidance on which approach should we pursue, considering we would like to be using for streaming inference. If any of those options are not optimal, could you suggest an alternative?