Baking model into Dockerimage

Justin · 2025-02-17T05:33:57.219Z

Hello, im trying to bake or rather, downlaod the model via vllm directly while building so the image contains the model. I havent found any kind of simple "vllm download" command sadly. The onlny was is either by running vllm and afterwards adding the file to the image which would be too big to host on my registry or let runpod serverless build the image for me with its doing it while building

3

3WaD•2/17/25, 6:21 AM

Why via vllm? You can do this step in the Dockerfile

33WaD Why via vllm? You can do this step in the Dockerfile

J

JustinOP•2/17/25, 7:16 AM

wdym? I am not sure how i can prewarm vllm viwthout using vllm to download it as its doing more than just downloding it from huggingface. Im not sure if it needs more but im also unsure how exaxctly to do otherwise

3

3WaD•2/17/25, 7:30 AM

If you want to bake a model into the container image you have to do it while building it. You can't do it afterwards without rebuilding it again, which can't be done on RunPod.

To download or copy a model and bake it into the image you can do:

# Use COPY for local files on your disk 
COPY localfolder/model /containerfolder/model

# Use COPY for local files on your disk 
COPY localfolder/model /containerfolder/model

# Use COPY for local files on your disk 
COPY localfolder/model /containerfolder/model

# Use COPY for local files on your disk 
COPY localfolder/model /containerfolder/model

or

#!/bin/bash
set -Eeuox pipefail

mkdir -p "$1"
cd "$1"
git init
git remote add origin "$2"
git fetch origin "$3" --depth=1
git reset --hard "$3"

#!/bin/bash
set -Eeuox pipefail

mkdir -p "$1"
cd "$1"
git init
git remote add origin "$2"
git fetch origin "$3" --depth=1
git reset --hard "$3"

#!/bin/bash
set -Eeuox pipefail

mkdir -p "$1"
cd "$1"
git init
git remote add origin "$2"
git fetch origin "$3" --depth=1
git reset --hard "$3"

#!/bin/bash
set -Eeuox pipefail

mkdir -p "$1"
cd "$1"
git init
git remote add origin "$2"
git fetch origin "$3" --depth=1
git reset --hard "$3"

# Make sure to have git installed!
COPY clone.sh /clone.sh

# Clone selected HuggingFace repo
RUN . /clone.sh /containerfolder/ https://huggingface.co/user/model

# Make sure to have git installed!
COPY clone.sh /clone.sh

# Clone selected HuggingFace repo
RUN . /clone.sh /containerfolder/ https://huggingface.co/user/model

# Make sure to have git installed!
COPY clone.sh /clone.sh

# Clone selected HuggingFace repo
RUN . /clone.sh /containerfolder/ https://huggingface.co/user/model

# Make sure to have git installed!
COPY clone.sh /clone.sh

# Clone selected HuggingFace repo
RUN . /clone.sh /containerfolder/ https://huggingface.co/user/model

J

JustinOP•2/17/25, 8:31 AM

If you want to bake a model into the container image you have to do it while building it. You can't do it afterwards without rebuilding it again, which can't be done on RunPod.

Exactly thats what i mean, my first thought would simply be using vvlm as its building the neccessary fodler structure but i think you can just skip this by using --model and the local path with that you sent right? will test it out thanks ^^

R

riverfog7•2/22/25, 6:11 AM

vllm --model /path/to/model

vllm --model /path/to/model

vllm --model /path/to/model

vllm --model /path/to/model does not work.
You have to do

vllm /path/to/model

vllm /path/to/model

vllm /path/to/model

vllm /path/to/model

Baking model into Dockerimage

Similar Threads

Similar Threads

Similar Threads