I have a serverless endpoint deployed with an idle timeout of 5 seconds. I expect that after 5 seconds, if I send a new request, the Docker image is downloaded again. Instead, the idle timeout is much longer. Even after minutes pass following sending a request again, the Docker image is already loaded, resulting in a very quick response which is good but i dont understand.
Continue the conversation
Join the Discord to ask follow-up questions and connect with the community
R
Runpod
We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!