Serverless vllm - lora
Is there a way to set the lora-modules (for the vllm docker container --lora-modules lora_adapter1=abc/efg) in the Template, or do i need to use the "standard" vllm container for it?
JIs there an option in the ui?
Sthere are only the options for enabling it and so on, the option for adding the url is missing
JDo you have the docs for the args
JIn vllm
JDocs
SFor the runpod vllm serverless worker, these are the ENVs, which are available: https://docs.runpod.io/serverless/workers/vllm/environment-variables
Configure your vLLM Worker with environment variables to control model selection, access credentials, and operational parameters for optimal performance. This guide provides a reference for CUDA versions, image tags, and environment variable settings for model-specific configurations.
JNo the one you wanna use, in vllm's docs
JMake an issue in the github of runpod vllm-worker I guess
HThanks Sven for making this
crazy enable_lora flag was added but not the arg to actually add lora adapters
crazy enable_lora flag was added but not the arg to actually add lora adapters
HHoping this gets merged soon
SThe Pull Request was merged. Does someone know when the new Versions appears for Runpod? (also the vllm worker with the 1.7.4 SDK?)
