LLM inference on serverless solution

Hi, need some suggestion on serving LLM model on serverless. I have several questions:
1. Is there any guide or example project I can follow so that can infer effectively on runpod serverless?
2. Is it recommended to use frameworks like TGI or vLLM with runpod? If so why? I'd like maximum control on the inference code so I have not tried any of those frameworks
Thanks!

ashley•2/20/24, 10:22 AM

RunPod have created a vLLM worker that you can use for serverless:
https://github.com/runpod-workers/worker-vllm

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

ribbitOP•2/20/24, 10:22 AM

Thanks! but I heard that vllm does not support quantized models?

ribbitOP•2/20/24, 10:26 AM

One of the reason I'd want maximum control to the inference code is that I want to run quantized models on other lib than transformers (exllama, etc.)

ashley•2/20/24, 10:26 AM

It does support some quantization types

ashley•2/20/24, 10:27 AM

Monosnap_runpod-workersworker-vllm_The_RunPod_worker_template_for_serving_our_large_language_model_endpoints._Powered_by_vLLM._2024-02-20_12-27-16.png

ribbitOP•2/20/24, 10:28 AM

I see, seems like exl is not yet supported, what are the real advantages of using vllm tho?

ashley•2/20/24, 10:31 AM

concurrency

ribbitOP•2/20/24, 11:48 AM

I see, will explore more on that, thanks!

LLM inference on serverless solution

Similar Threads

Similar Threads

Similar Threads