TheBloke/goliath-120b-GPTQ with RunPod Kobold AI United

Hi! I got goliath-120b-GPTQ running with 3 A40. But the text generation speed is extremely slow. What is the best option for GPU config and settings to run this model?

Thank you in advance!
Was this page helpful?