R
RunPod•23h ago
TF

(Beginner Question) Hosting Quantized model

Hi, I'm new to runpod can anyone point me towards how I can host a quantized model like this? I want to try the 2.71bit version first. https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF-UD
2 Replies
riverfog7
riverfog7•22h ago
4xA100 H100 Should work Try vllm with 4xA100 Its supported with runpod's official template
TF
TFOP•22h ago
Thank you I'll test it out and report back 🙂

Did you find this page helpful?