How to install ollama (and download models) in to /workspace?
I've got the pod working as expected but the ollama install.sh script installs to a default directory that I cannot figure out how to change. (/usr/local). This is of course not useful for runpod because this space gets reset every time the pod restarts. How do I install ollama and store models in to /workspace so that it persistent?
Solution:Jump to solution
If you just want to run LLM's check out https://get.runpod.io/koboldcpp if you want a way to run LLM's on runpod thats more optimized
6 Replies
You can set the
OLLAMA_MODELS variable. No idea why this isn't documented by ollama.
By default it's ~/.ollama/models
Oh I misread your request, unfortunately ollama doesn't let you change where it's installed to without also modifying their installation script. You could make a custom Docker Image that installs ollama ontop of our runpod/base
Sources:
https://github.com/ollama/ollama/issues/8640
https://github.com/ollama/ollama/pull/9646OK that makes sense why I was struggling 🙂 I was only using ollama because that is what the first suggestion was in the article i read about how to use runpod to host models... Is there a different approach that is more common and straight forward where I can choose which model to host and access via OpenAI compatible API endpoint?
Ollama is working but using a startup script to set it up from scratch every time the container boots feels like a bit of a bandaid workaround, im sure theres a better/proper way
Solution
If you just want to run LLM's check out https://get.runpod.io/koboldcpp if you want a way to run LLM's on runpod thats more optimized
Speaks OpenAI very well, even has some of ollama's api on board
Just give it the gguf in the environment variable before booting and its ready to go in notime
@Henky!! that looks perfect thank you!
Let me know if you need help with it