VLLM model loading, TTFT unhappy path
I am looking for a way to reduce the latency for the unhappy path of VLLM endpoints.
I use the quickstart VLLM template, backed by a network storage for model weights and flashboot enabled. By default the worker will load the model weights on first request. This, however poses the risk of exposing my customers to an unhappy path of latency measured in minutes, at scale we could see this in significant absolute numbers.
What would be the best way for me to make sure that a worker is considered ready only >after< it has loaded the model checkpoints, and trigger checkpoint loading without sending the first request? Should I roll my own VLLM container image? Or is there an idiomatic way to parametrize the quickstart template to achieve this? I would prefer to use the Runpod supplied, properly supported VLLM image, if possible....
Serverless github endpoint stuck at uploading phase
First of all i'd like to thank the Runpod team for their amazing work ! 🎉
Although it worked on the initial deployement, I seem to have an issue with subsequent ones.
The worker build and deploy correctly but the UI is stuck at the "uploading" phase and the active build doesn't update....
Serverless request returns None from python client but web status says completed successfully
Hello, I have been baffled by this issue for weeks and im pulling my hair out.
I have a serverless endpoint that always comes back as None from the python runpod client with no error messages in the logs or from my inference script.
Yet the runpod.io metrics for my requests always show has completed. ...
Template id missing in serverless dashboard
Just noticed the template id is completely missing from the endpoints. I have some templates with same docker image tag but different template id so that when I'm testing new images on the test endpoint it doesn't mess up the production endpoint. Can the template id pls be added back like next to docker image?
