© 2026 Hedgehog Software, LLC

Twitter GitHub Discord

More

Communities Docs About Terms Privacy

GGUF in serverless vLLM - Runpod

Runpod•2y ago•

57 replies

GGUF in serverless vLLM

How do I run a GGUF quantized model?
I need to run this LLM: https://huggingface.co/mradermacher/OpenBioLLM-Llama3-70B-GGUF

What parameters should I specify?

Thank you

mradermacher/OpenBioLLM-Llama3-70B-GGUF · Hugging Face

mradermacher/OpenBioLLM-Llama3-70B-GGUF · Hugging Face

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

21,906Members

Sponsored

Resources

Similar Threads

Was this page helpful?

Recent Announcements

Similar Threads

RRunpod / ⚡｜serverless

Serverless VLLM batching

RRunpod / ⚡｜serverless

Serverless vllm - lora

RRunpod / ⚡｜serverless

vLLM Serverless error

RRunpod / ⚡｜serverless