JohnTheNerd
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
I failed to figure it out
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
that quantization config looks wrong
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
that doesn't sound right
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
that should be way more than enough
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
good thing I have the 17$ on my account lol
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
I'll do that, yes
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
that's true
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
I should try it
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
interesting
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
maybe a support thread in the runpod discord isn't the best place to discuss this tho lol
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
AWQ is nice because it relies on calibration to determine the most important 1.5% of weights. then it leaves those at fp16, quantizing the rest to int4
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
throwing that all in
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
I'm thinking of just taking those scales and injecting them into the safetensors file for the awq quant
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
I have a different idea...
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
I could do that but I suspect it'll reduce quality significantly
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
just a pain
548 replies
RRunPod
•Created by JohnTheNerd on 4/7/2025 in #⛅｜pods-clusters
Pod ran out of CPU RAM
this can be extracted
548 replies