Unsloth Llama Scout will not download
Hi, I'm in Runpod in oobabooga one-click and I'm trying to run Unsloth Llama Scout Q6_K on 2x A40 (just for conversation, not training/learning). I've followed the directions listed at Unsloth's official "Llama 4: How to Run and Fine-Tune", but get stuck at step 2 of "how to run", where it says to select what model you would like to use. In the coding given at Unsloth's page, I put allow_patterns = "Q6_K", and get told that there's no such command.
I had originally tried downloading it using the regular oobabooga interface. It "downloads" very quickly, but there's nothing actually there. When I try to load the model using llamaccp, I get told "list index out of range". The same thing happened with wget in the terminal. It was 127 kb.
I'm completely new at this and really have no idea what I'm doing. I had been getting Google Gemini to help me but that wasn't going anywhere. I'm grateful for any and all help or feedback. Thank you.




1 Reply
I recommend switching to https://koboldai.org/runpodcpp which is optimized to run GGUF on Runpod
Simply edit the template, expand the variable and replace the KCPP_MODEL link with a link to the gguf file you wish to run, should be easy from there
Although I'm getting flashbacks so if we already did that its solved haha