Can I use LoRA in vLLM serverless with OpenAI API?
I need both LoRA and Structured Outputs, but it seems like LoRA is only supported by Runpod API and Structured Outputs are only (poorly) supported by OpenAI API? 
P
JAre you using vllm worker?
Jlora should be able to be load in the configuration via environment variable if i remember correctly, and whats the problem with structured output in openai api?
DYes
DI think environment variable is possible but only for hugging face not for Runpod storage?
DStructured output does work for OpenAI API but does not enforce valid json 100% and does not enforce json field order
DSo I'm using guided regex with some post processing instead
JDoes your model truly supports. Structured outputs?
DIs not model specific
JI mean env variable to configure vllm to also load loras
DYes but the LoRA itself needs to be on hugging face?
JI think so
JOr it can be cached locally but you'll have to add it to your custom image manually
DDDJWhat reason are you attaching those links for?
Jhttps://github.com/runpod-workers/worker-vllm/blob/main/docs/configuration.md
Is this not working or?
Is this not working or?
GitHub
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
