llama.cpp serverless endpoint
https://github.com/ggerganov/llama.cpp
llama.cpp is afak the only setup that supports llava-1.6 quantized, that's why i use it.
On some workers the docker image works, on others "illegal instruction" error and crash.
https://github.com/ggerganov/llama.cpp/issues/537
I wonder if someone already tried it out and if there's a better fix to this issue other than building and stuffing multiple binaries with the correct instruction sets into one image that will work anywhere. (i already tried building with LLAMA_NATIVE=0)
appreciate any insights, thanks!
llama.cpp is afak the only setup that supports llava-1.6 quantized, that's why i use it.
On some workers the docker image works, on others "illegal instruction" error and crash.
https://github.com/ggerganov/llama.cpp/issues/537
I wonder if someone already tried it out and if there's a better fix to this issue other than building and stuffing multiple binaries with the correct instruction sets into one image that will work anywhere. (i already tried building with LLAMA_NATIVE=0)
appreciate any insights, thanks!
GitHub
Port of Facebook's LLaMA model in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.
GitHub
I try to make it run the docker version on Unraid, I run this as post Arguments: --run -m /models/7B/ggml-model-q4_0.bin -p "This is a test" -n 512 I got this error: /app/.devops/tools.sh...
Solution
I don't know why you would want to use llama.cpp, its more for offloading onto CPU than for GPU. You can look at using this instead:
https://github.com/ashleykleynhans/runpod-worker-llava
https://github.com/ashleykleynhans/runpod-worker-llava
GitHub
LLaVA: Large Language and Vision Assistant | RunPod Serverless Worker - ashleykleynhans/runpod-worker-llava
