Access to remote storage from vLLM
I want to make an API call with a file that is on a my RunPod remote storage.
But vLLM tell me:
Cannot load local files without --allowed-local-media-path
I've tried to set a an ENV variable `ALLOWED_LOCAL_MEDIA_PATH = true but it's not working.
I know I can try with a S3 remote storage or even a BASE64 but it's not as efficient as a mounted remote storage.Solution:Jump to solution
It works with that (if it can help others):
"allowed_local_media_path": os.getenv('ALLOWED_LOCAL_MEDIA_PATH', '/runpod-volume')
Add this line in:
/worker-vllm/src/engine_args.py
So you can add an ENV variable with the paths you want (or by default it will be/runpod-volume)....9 Replies
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
I need to use a Serveless Endpoint. I've build my own docker image with https://github.com/runpod-workers/worker-vllm
But this NEV variable is not listed on the available ENV on the doc.
I think i'll have to modify this file: https://github.com/runpod-workers/worker-vllm/blob/main/src/engine_args.py
And add ALLOWED_LOCAL_MEDIA_PATH.
GitHub
GitHub - runpod-workers/worker-vllm: The RunPod worker template for...
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
GitHub
worker-vllm/src/engine_args.py at main · runpod-workers/worker-vllm
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
Yes i'll try to add that:
"allowed-local-media-path": os.getenv('ALLOWED_LOCAL_MEDIA_PATH', '/runpod-volume')
Still got
"Cannot load local files without --allowed-local-media-path. None"
😢
Maybe this param need to be set to true?Unknown User•2w ago
Message Not Public
Sign In & Join Server To View
yes thanks!
i'm gonna try it
Solution
It works with that (if it can help others):
"allowed_local_media_path": os.getenv('ALLOWED_LOCAL_MEDIA_PATH', '/runpod-volume')
Add this line in:
/worker-vllm/src/engine_args.py
So you can add an ENV variable with the paths you want (or by default it will be/runpod-volume).Maybe it should be added in worker-vllm.
Will make a PR if I have time.
Unknown User•2w ago
Message Not Public
Sign In & Join Server To View