Runpod•2y ago•

57 replies

GGUF in serverless vLLM

How do I run a GGUF quantized model?
I need to run this LLM: https://huggingface.co/mradermacher/OpenBioLLM-Llama3-70B-GGUF

What parameters should I specify?

Thank you

mradermacher/OpenBioLLM-Llama3-70B-GGUF · Hugging Face

digigoblin•5/30/24, 11:57 AM

You will have to create your own serverless handler for it, because the vllm worker does not support GGUF due to the underlying vllm engine not supporting GGUF.

ArmykOP•5/30/24, 11:58 AM

Such a shame that it doesn't. Can I run Ollama in serverless?

digigoblin•5/30/24, 11:58 AM

You can run whatever you want in serverless as long as you implement the RunPod serverless handler

AArmyk Such a shame that it doesn't. Can I run Ollama in serverless?

Jason•5/30/24, 11:59 AM

Is there any template that supports it? if yes -> it can and use it
if not, it still can but make your own template

ArmykOP•5/30/24, 12:01 PM

By templates, do you mean the very limited ones from "quick deploy", or any template that can be run in a normal pod like on the screenshot? I can't input the ollama/ollama template on the serverless deployment

digigoblin•5/30/24, 12:02 PM

You can't use pod templates in serverless, they don't work the same way, you need to invoke the serverless handler for serverless as I mentioned above.

ArmykOP•5/30/24, 12:03 PM

Where can I browse community templates for serverless? There has to be someone that already did this

JJason Is there any template that supports it? if yes -> it can and use it if not, it s...

digigoblin•5/30/24, 12:03 PM

Not sure why you asking this, serverless templates aren't shared publicly like pod ones.

digigoblin•5/30/24, 12:04 PM

The only serverless template available is the vllm one that RunPod created.

AArmyk Where can I browse community templates for serverless? There has to be someone t...

digigoblin•5/30/24, 12:04 PM

You can't. @nerdylive must have been confused.

ArmykOP•5/30/24, 12:06 PM

⚡｜serverlessHow to run OLLAMA on Runpod Serverless?

ArmykOP•5/30/24, 12:06 PM

I found this thread. I will probably need to configure it myself. Thank you for your help.

Jason•5/30/24, 12:08 PM

yep there are no place for sharing community templates for serverless yet on the site

digigoblin•5/30/24, 12:21 PM

@Papa Madiator has this list of Open Source things tho:
https://github.com/kodxana/Awesome-RunPod

GitHub

GitHub - kodxana/Awesome-RunPod: A curated list of amazing RunPod p...

A curated list of amazing RunPod projects, libraries, and resources - kodxana/Awesome-RunPod

AArmyk https://discord.com/channels/912829806415085598/1221249312495898675

Madiator2011•5/30/24, 12:21 PM

I made template with open web ui

MMadiator2011 I made template with open web ui

digigoblin•5/30/24, 12:22 PM

For pod tho, not serverless isn't it? he is looking for serverless solution.

Madiator2011•5/30/24, 12:22 PM

AArmyk Such a shame that it doesn't. Can I run Ollama in serverless?

PatrickR•5/31/24, 2:41 PM

https://docs.runpod.io/tutorials/serverless/cpu/run-ollama-inference

Run an Ollama Server on a RunPod CPU | RunPod Documentation

Learn to set up and run an Ollama server on RunPod CPU for inference with this step-by-step tutorial.

ArmykOP•5/31/24, 3:41 PM

Cpu inference isn't good enough but thank you

AArmyk Cpu inference isn't good enough but thank you

Jason•5/31/24, 3:42 PM

You can also do it with GPU view: How to run OLLAMA on Runpod Serverless?

PatrickR•5/31/24, 7:16 PM

@Armyk If you follow that guide, but just select GPU, you’ll get the same results.

Alpay Ariyak•6/4/24, 6:43 PM

https://github.com/vllm-project/vllm/pull/5191

GitHub

[Core] Support loading GGUF model by Isotr0py · Pull Request #5191 ...

FILL IN THE PR DESCRIPTION HERE
Related issue: #1002
Features:

This PR adds support for loading GGUF format model
This PR will also add gguf to requirements.
Currently, only llama is modified for ...

Alpay Ariyak•6/4/24, 6:44 PM

Seems vLLM will have GGUF support soon!

guestavius•6/19/24, 3:41 PM

I just created an account today and am looking at the serverless vLLM quick deploy settings. If GGUF isn't supported, what's this thing? I don't see a bpw / quant level setting.

Gguestavius I just created an account today and am looking at the serverless vLLM quick depl...

digigoblin•6/19/24, 3:42 PM

VLLM engine does not support GGUF, see the messages above.

guestavius•6/19/24, 3:42 PM

I'm asking what that drop menu do.

digigoblin•6/19/24, 3:43 PM

It does what it says, select quantization type

digigoblin•6/19/24, 3:44 PM

Its not quant level, its quant TYPE

digigoblin•6/19/24, 3:46 PM

Docs are available here:
https://github.com/runpod-workers/worker-vllm

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

guestavius•6/19/24, 3:51 PM

[redacted] ~~I'm hoping not to need 360GB of VRAM to run an 8x22B.~~
Edit: Oh wait, that just means I can point a name/model-AWQ-or-GPTQ repository to serverless.

guestavius•6/19/24, 6:31 PM

>two
looks like I wouldn't be able to run it anyway
Edit: I'll have to quantize this obscure model...

digigoblin•6/19/24, 6:45 PM

Which model? Many models are already available as quantized versions

guestavius•6/19/24, 6:48 PM

https://huggingface.co/gghfez/WizardLM-2-8x22B-Beige
This one caught my interest. No idea if it's good though.

digigoblin•6/19/24, 6:54 PM

I see there is EXL2 quantized versions but vllm doesn't support EXL2 quant type

digigoblin•6/19/24, 6:54 PM

Aphrodite Engine and TabbyAPI both support EXL2 tho.

Charixfox•6/20/24, 2:08 AM

Until vLLM supports more quant formats, you'll have to have an AWQ, SqueezeLLM, or GPTQ quant of the model. I used a Jupyter pod to make an AWQ of the model I wanted. Or if Aphrodite-Engine ever works on serverless, that will be an option too.

guestavius•6/20/24, 2:23 AM

I ended up using KoboldCpp's runpod template for gguf, lol. And sharing with some people to spend less time idling. (I'm being an idiot, yes.)

Charixfox•6/20/24, 3:41 AM

If all else fails, just run the numbers to see if serverless will be better for your use cases. There's an amount of active time where pods become more cost efficient.

Charixfox•6/20/24, 3:54 AM

On last gen 48GB for example, it's 40% active runtime (Actively processing requests)

Hermann•6/22/24, 2:55 PM

GGUF is a format for offline use on your own computer. It's not meant to be for servers really. Use AWQ or GPTQ untill ex2 is supported on vLLM.

digigoblin•6/22/24, 3:14 PM

GGUF supports both CPU and GPU not just CPU

Hermann•6/22/24, 3:34 PM

Yeah and It's not the fastest though.

digigoblin•6/22/24, 3:45 PM

EXL2 is fastest.

Hermann•6/23/24, 3:12 PM

Yeah I just hope vLLM would one day support EXL2. It would open up so many new opportunities.

digigoblin•6/23/24, 3:51 PM

aphrodite engine supports it

Hermann•6/24/24, 8:40 AM

Yes, but aphrodite runs only on classic Pods and it's very expensive to run.

This is why I love serverless, it's cheap to begin with (but gets 3 x more expensive if you have constant traffic). Serverless is great to start a project with minimal traffic, only if the project is a success and can generate money, then it's worth it to switch to a classic pod with aphrodite.

digigoblin•6/24/24, 8:48 AM

You can port aphrodite to serverless too

Hermann•6/24/24, 4:38 PM

But it will be experimental right? I don't think it's that easy and I'm so busy as it is with coding.

HHermann But it will be experimental right? I don't think it's that easy and I'm so busy...

Jason•6/24/24, 4:56 PM

What's experimental

digigoblin•6/24/24, 5:03 PM

He is probably referring to aphrodite-engine. Its not experimental, TabbyAPI is:
"TabbyAPI is a hobby project solely for a small amount of users. It is not meant to run on production servers. For that, please look at other backends that support those workloads."

GGUF in serverless vLLM

Similar Threads

Similar Threads

Similar Threads