R
RunPodā€¢6mo ago
EngrBilal.

What happens during cold start time?

Hi! I am new to docker and as well as to serverless. I would like to know what happens during cold start, does my image gets pulled again each time? Or it's just the time required for provising resources and loading models in vram etc?
22 Replies
nerdylive
nerdyliveā€¢6mo ago
Yes, the second one Image pulling(exc from cold start) in serverless wont be billed, cold start will
EngrBilal.
EngrBilal.OPā€¢6mo ago
Awesome. I have another question. I have model weights of 30gbs. Do you recommend adding them in docker image or I should download them using a script to mounted volume?
nerdylive
nerdyliveā€¢6mo ago
download them into a mounted volume
EngrBilal.
EngrBilal.OPā€¢6mo ago
What's the path of mounted drive?
nerdylive
nerdyliveā€¢6mo ago
usually on serverless its /runpod-volume but if on pods its usually /workspace
EngrBilal.
EngrBilal.OPā€¢6mo ago
Cool. Thanks for answers
yhlong00000
yhlong00000ā€¢6mo ago
I think flash recommended to add your model a part of docker image whenever possible. It probably better than network volume for performancešŸ˜€
EngrBilal.
EngrBilal.OPā€¢6mo ago
I am super confused now. Lol
nerdylive
nerdyliveā€¢6mo ago
What's up
EngrBilal.
EngrBilal.OPā€¢6mo ago
How to decide when to add model as part of docker image or download it on mounted drive
nerdylive
nerdyliveā€¢6mo ago
I mean if you want you can test them to decide Which suits better And time your tests Lmk if you did that hahah Your model is quite big right
EngrBilal.
EngrBilal.OPā€¢6mo ago
30gbs of disk space
nerdylive
nerdyliveā€¢6mo ago
Yeah Try them both, and run tests
EngrBilal.
EngrBilal.OPā€¢6mo ago
Image is being pulled on each cold start
nerdylive
nerdyliveā€¢6mo ago
No its just the loading of your model, applications No pulling again Check the logs
yhlong00000
yhlong00000ā€¢6mo ago
If you think about it, when your Docker image contains everything necessary, the container is ready to go as soon as it starts, with all data stored on the host disk for fast access. In contrast, if you store the model on a network volume, you would need to mount it and connect through the data centerā€™s Ethernet to network storage, which is likely to be slower than accessing the local disk.
EngrBilal.
EngrBilal.OPā€¢6mo ago
I checked on my docker hub, I can see how many times my image was being pulled
nerdylive
nerdyliveā€¢6mo ago
Okay the logs also say when it's pulled
yhlong00000
yhlong00000ā€¢6mo ago
When deploying your serverless function for the first time requires pulling the image, which can be slow. If you keep sending requests, the container remains active, that avoid cold starts. However, if thereā€™s a pause in requests, Runpod stops the container. When requests resume, a cold start occurs, but it will be faster than the initial image pull.
nerdylive
nerdyliveā€¢6mo ago
Yeah cold starts aren't loading image for the first time, cold starts only for when you rarely hit requests to your endpoint
frank
frankā€¢4mo ago
@nerdylive How long before the endpoint goes ā€œcoldā€? It doesn't seem to be a constant time, if so do you know what it is?
nerdylive
nerdyliveā€¢4mo ago
yeah no specific amount
Want results from more Discord servers?
Add your server