Serverless vLLM changing engine arguments
Hi, I got vLLM Serverless worker up and running, but want to change one engine argument (which is not overridable through environment variables), specifically
--limit-mm-per-prompt , how could I do that with your custom image runpod/worker-v1-vllm:v2.3.0stable-cuda12.1.0 that endpoints use? Thanks7 Replies
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
thanks I'll look into it and report back
@Jason https://github.com/runpod-workers/worker-vllm/issues/155 looks like support for it was implemented, and then someone accidentally removed it lol
Would it be possible to add it back again?
GitHub
setting limit_mm_per_prompt · Issue #155 · runpod-workers/worker-...
Is it possible to set limit_mm_per_prompt? there as a PR about it but this feature is not in main.
Created PR to add it back https://github.com/runpod-workers/worker-vllm/pull/177
GitHub
fix: added back limit_mm_per_prompt to engine args by aleksandar-ba...
Added back the engine argument limit_mm_per_prompt that was accidentally removed as per #155
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View
@Jason Thanks!
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View