What happens during cold start time?
Hi! I am new to docker and as well as to serverless. I would like to know what happens during cold start, does my image gets pulled again each time? Or it's just the time required for provising resources and loading models in vram etc?
22 Replies
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
Awesome. I have another question. I have model weights of 30gbs. Do you recommend adding them in docker image or I should download them using a script to mounted volume?
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
What's the path of mounted drive?
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
Cool. Thanks for answers
I think flash recommended to add your model a part of docker image whenever possible. It probably better than network volume for performanceš
I am super confused now. Lol
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
How to decide when to add model as part of docker image or download it on mounted drive
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
30gbs of disk space
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
Image is being pulled on each cold start
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
If you think about it, when your Docker image contains everything necessary, the container is ready to go as soon as it starts, with all data stored on the host disk for fast access. In contrast, if you store the model on a network volume, you would need to mount it and connect through the data centerās Ethernet to network storage, which is likely to be slower than accessing the local disk.
I checked on my docker hub, I can see how many times my image was being pulled
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
When deploying your serverless function for the first time requires pulling the image, which can be slow. If you keep sending requests, the container remains active, that avoid cold starts. However, if thereās a pause in requests, Runpod stops the container. When requests resume, a cold start occurs, but it will be faster than the initial image pull.
Unknown Userā¢17mo ago
Message Not Public
Sign In & Join Server To View
@nerdylive How long before the endpoint goes ācoldā? It doesn't seem to be a constant time, if so do you know what it is?
Unknown Userā¢15mo ago
Message Not Public
Sign In & Join Server To View