is AWQ faster than GGUF ?

In which order is the fastest inference speed between AWQ, GGUF, GPTQ, QAT, EXL2 ?

digigoblin•4/17/24, 5:05 PM

EXL2

VolkoOP•4/17/24, 5:13 PM

Okay thanks

digigoblin•4/17/24, 5:15 PM

Thats the fastest, don't know about the others, never actually used QAT or GGUF.
@aikitoria will probably know.

aikitoria•4/18/24, 7:27 PM

I've not used AWQ or GPTQ directly, those are older formats

aikitoria•4/18/24, 7:27 PM

you use GGUF if you want to run on a very small GPU and have to keep some of the model on CPU only. it's for hybrid CPU/GPU inference

aikitoria•4/18/24, 7:27 PM

you use EXL2 for maximum speed on a single GPU

aikitoria•4/18/24, 7:28 PM

you use aphrodite-engine or TensorRT-LLM (good luck!) for maximum speed on multiple GPUs

Geri•7/12/24, 8:53 PM

hi do you use tensorrt llm?