Serverless vllm - lora
Is there a way to set the lora-modules (for the vllm docker container --lora-modules lora_adapter1=abc/efg) in the Template, or do i need to use the "standard" vllm container for it?
7 Replies
Unknown User•14mo ago
Message Not Public
Sign In & Join Server To View
there are only the options for enabling it and so on, the option for adding the url is missing
Unknown User•14mo ago
Message Not Public
Sign In & Join Server To View
For the runpod vllm serverless worker, these are the ENVs, which are available: https://docs.runpod.io/serverless/workers/vllm/environment-variables
Environment variables | RunPod Documentation
Configure your vLLM Worker with environment variables to control model selection, access credentials, and operational parameters for optimal performance. This guide provides a reference for CUDA versions, image tags, and environment variable settings for model-specific configurations.
Unknown User•14mo ago
Message Not Public
Sign In & Join Server To View
Thanks Sven for making this
crazy enable_lora flag was added but not the arg to actually add lora adapters
Hoping this gets merged soon
The Pull Request was merged. Does someone know when the new Versions appears for Runpod? (also the vllm worker with the 1.7.4 SDK?)