Can we run aphrodite-engine on Serverless?
aphrodite-engine is a fork from vLLM and also supports exl2 format, which gives it a huge advantage. Are there any plans to support aphrodite-engine in future on RunPod's serverless offering?
I believe currently aphrodite-engine is only supported as a single server on RunPod.
Thanks
8 Replies
I mean nothing stops your from building own worker 🙂
How? lol
all workers are open source so you can have look at code and build own. Some coding and then packaging docker image
Seriously, I would be interested building my own queue and host aphrodite-engine on a single pod instead.
You mean this worker: https://github.com/runpod-workers/worker-vllm
GitHub
GitHub - runpod-workers/worker-vllm: The RunPod worker template for...
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
Or any other you could recommend?
I do not know what aphrodite-engine is so cant tell but yes if it's fork ov vllm it will be good start point
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View