How do I set quantization to fp8 in the serverless settings?
If I select 'bitsandbytes' will that automatically change quant level to fp8?
3 Replies
Unknown User•3mo ago
Message Not Public
Sign In & Join Server To View
What do you mean?
I want to force it to use fp4/8 etc
Unknown User•3mo ago
Message Not Public
Sign In & Join Server To View