Model loading time
i'm running a custom image that includes mixtral 8*7B model where model loading itself is taking 4.5 minutes to load each time a worker starts. i don't want to use an active worker given my load and also do not want to waste 4.5 minutes for model loading as that incurs me additional cost. any better alternatives for this. I already cache models and use network volume with the serverless endpoint.
0 Replies