eric.mattmann
RRunPod
•Created by c on 5/9/2025 in #⚡｜serverless
Veryyyyyy slow serverless VLLM
Hi !
Using a network volume to store the .cache folder is tempting, it could speed up vllm startup once most of the hardware configurations have been cycled through.
But a network volume can only be shared at a single location, so it limits your choices.
I believe those vllm/torch caches are quite small : do you know if it can be attempted to sync them from another network location than a shared volume? Doing this asynchronously at endpoint startup and then once it has warmed up could work…
50 replies
RRunPod
•Created by eric.mattmann on 3/19/2025 in #⚡｜serverless
Error 404 on payload download.
It was actually that the file was not yet made available on our end. Runpod rocks! Thank you for your answer.
4 replies