R
RunPod•3mo ago
Thick Thighs

The Bloke LLM Template ExLlamaV2Cache_Q4 Error

Has anyone found a way around this. I use to use the pip install --upgrade exllamav2 command in the terminal but now that doesn't work. It worked yesterday but I'm guessing some things have changed and now it doesn't. The issue from what it seems has been going on for 2 or so weeks judging by the issues tab in github. https://github.com/TheBlokeAI/dockerLLM/issues/17. Using pip install --upgrade --no-deps exllamav2 solves it for now but that is only temporary I'm wondering if anyone has a updated version of the blokes template that can permanently solve this.
GitHub
Issues · TheBlokeAI/dockerLLM
TheBloke's Dockerfiles. Contribute to TheBlokeAI/dockerLLM development by creating an account on GitHub.
2 Replies
ashleyk
ashleyk•3mo ago
TheBloke has disappeared for almost 2 months, you can use my template in the meantime.
Thick Thighs
Thick Thighs•3mo ago
Thank you so much!
Want results from more Discord servers?
Add your server
More Posts
Hello, I have a docker image downloaded on to the pod. How to I use my custom image?I don't think I can do a docker inside a docker, can I? I appreciate any guidance here!Machine does not support exposing a TCP portMy prod pod needs to expose public network ports, so we used the template to configure "Expose TCP pCannot Install JAXHello, I am currently unable to properly install JAX on both the A100 SXM 80GB and the H100 80GB SXMGPU Name"NVIDIA RTX 4000 Ada Gene..."GPU 0"Error: CUDA unknown error - this may be due to anPOD ID: 86gy23dlljmxvjhow to get kernel 5.5.0?trying to get newer kernel to fine-tune llm. thanksServerless prod cannot import name "ControlNetModel"When I 'runpodctl project dev', it run well in dev. But when I 'runpodctl project deploy'，run, it sh