How to cache model download from HuggingFace - Tips?
Usin Serverless (48gb pro) w Flashboot. Want to optimize for fast cold start
is there a guide somewhere?
it does not seem to be caching the download - it's always re-downloading the model entirely (and slowly)
should i ssh into some persistent storage & download the model there? then reference that local path in the HF model load?
9 Replies
Unknown User•9mo ago
Message Not Public
Sign In & Join Server To View
@nerdylive would u recommend doing this (pic) ? (seems all workers in my endpoint will pull from this same /runpod-volume)
-
btw: perhaps runpod-volume is only available/mounted when when using a runpod docker base image? e.g. a ubuntu image doesn't seem to have it mounted (pic))
also: it seems like when you change GPU type, the /runpod-volume is deleted/non accessible - is this correct?
Unknown User•9mo ago
Message Not Public
Sign In & Join Server To View
okay thanks. do you recommend creating a new network volume? & persisting HF weights in that?
perhaps that's more stable/clear for me to follow than using the default /runpod-volume (which i assume is attached by default?) but seems to be giving me unexpected behaviour
------
i seem to be triggering new HF downloads even when this image has run & downloaded & persisted the weights to /runpod-volume/.cache/huggingface/hub/.. in previous runs
Downloading shards: 0%| | 0/3 [00:00<?, ?it/s]Request 48c80db3-d744-4f39-8af2-929133a77895: HEAD https://huggingface.co/LanguageBind/Video-LLaVA-7B
if u happen to know / have a code example that shows a reliable way to persist HF in he most straightforward way lmk!
Unknown User•9mo ago
Message Not Public
Sign In & Join Server To View
when writing to /rundpod-volume i'm still seeing the container do full model downloads when i kill the worker
so i:
- created a new network storage ( /modelstorage ) & and am read/writing to this
- attached this volume to my endpoint (didn't deploy the volume)
but when i kill the worker it re-downloads from hf??
any am i missing !? any code examples of ensuring it downloads from the network volume & NOT hf
Unknown User•9mo ago
Message Not Public
Sign In & Join Server To View