Runpod•2y ago•

49 replies

Run LLM Model on Runpod Serverless

Hi There,

I have LLM Model which build on docker image and it was 40GB++ docker Image.

I'm wondering, can I mount the model as volume instead of add the model in the docker image?

Thanks !

ashley•2/16/24, 10:17 AM

Yes, you can put your model on network storage and load it from there, but its generally more performant to bake the model into the docker image because network storage is incredibly slow. Network storage also limits GPU availability.

TumbleWeedOP•2/16/24, 10:18 AM

Is that okay to bake the model into the docker image @ashleyk ?
does it affected the cold start?

ashley•2/16/24, 10:18 AM

No, loading it from network storage affects cold start more.

TumbleWeedOP•2/16/24, 10:19 AM

How about the pulling image strategy, does the runpod cache the image to internal registry, or it would pull the image everytime the worker spawned?

ashley•2/16/24, 10:19 AM

Your docker image is cached onto the workers in advance so has no impact on cold start times.

TumbleWeedOP•2/16/24, 10:20 AM

Alright thank you, I will try it first

TumbleWeedOP•2/16/24, 11:55 AM

@ashleyk I have tried to setup the serverless endpoint, how to check the logs of the pull?
how do I know if the worker success pull the image

ashley•2/16/24, 11:56 AM

Click on each worker and check. The workers will go "Idle" when they are done pulling the image.

TumbleWeedOP•2/16/24, 11:59 AM

It stucked on "Initializing"

TumbleWeedOP•2/16/24, 12:00 PM

Does it returned error if the pull image fail?
let say I have missconfigured the registry access

ashley•2/16/24, 12:01 PM

Click on the workers to check the logs.

TumbleWeedOP•2/16/24, 12:02 PM

I see, does I got charged when the worker is on initializing state?

TTumbleWeed I see, does I got charged when the worker is on initializing state?

ashley•2/16/24, 12:04 PM

No, only while the container is running - cold start + execution time.

TumbleWeedOP•2/16/24, 12:04 PM

Wow, okay

TTumbleWeed Wow, okay

J.•2/16/24, 4:06 PM

If u can share the screenshot of ur template also be good

J.•2/16/24, 4:06 PM

sometimes ppl forget the tag

J.•2/16/24, 4:06 PM

so just double check

J.•2/16/24, 4:06 PM

such as

J.•2/16/24, 4:06 PM

username/image:1.0

J.•2/16/24, 4:06 PM

some ppl just write

J.•2/16/24, 4:06 PM

username/image

Alpay Ariyak•2/16/24, 10:11 PM

Just use our pre-made worker vLLM image and attach a network volume

Alpay Ariyak•2/16/24, 10:11 PM

On startup, the worker will download the model to the network storage, and all the workers will have access to it

Alpay Ariyak•2/16/24, 10:11 PM

The image itself is only 3gb as well and no need to build it

TumbleWeedOP•2/17/24, 1:36 PM

I have sucessfully run my model, but need some adjustment, because inside the container, I still run the FastAPI for endpoint.

ashley•2/17/24, 1:50 PM

Yeah you don't need FastAPI for serverless, serverless already provides an API layer for you.

TumbleWeedOP•2/19/24, 7:47 AM

Hi @ashleyk @Alpay Ariyak
I have tried deploying my LLM Model on runpod serverless, the image is over 40GB, it doesn't cost wise to use Google Artifact Registry as they charging for egress outside of GCP Network, any recommendations for the container registry?

Thank you

ashley•2/19/24, 7:49 AM

I just use Dockerhub, its free

TumbleWeedOP•2/19/24, 7:50 AM

But they limit the pull requests right?

ashley•2/19/24, 7:50 AM

Yeah they have rate limiting by IP, but you can use your token to authenticate instead

ashley•2/19/24, 7:51 AM

I use dockerhub in my production serverless endpoints and never had any issues.

TTumbleWeed But they limit the pull requests right?

J.•2/19/24, 7:51 AM

I was testing Llama models / mistral models that are close to the 35Gb/40Gb mark through dockerhub no issues - u can add ur docker credentials to runpod too

Also dockerhub has one private repo per account - if u have it as something sensitive; which then obvs need to add ur docker credential to runpod settings

TumbleWeedOP•2/19/24, 7:53 AM

Alright, I will try with dockerhub then, Thank you @ashleyk @justin

TTumbleWeed Hi @ashleyk @Alpay Ariyak I have tried deploying my LLM Model on runpod serverl...

J.•2/19/24, 7:53 AM

Cant believe google charges u egress for a registry

J.•2/19/24, 7:53 AM

i know they do for gcp bucket data but rlly for a container registry? damn

ashley•2/19/24, 7:56 AM

Google, AWS and Azure all charge massive egress costs for everything

TumbleWeedOP•2/19/24, 10:55 AM

I have move my LLM model to Dockerhub, so I don't get haunted by gcp egress cost lol

I have another question
my cold start (load the LLM Model ) is around 15s-30s, anyway to optimize it? @ashleyk @justin

TTumbleWeed I have move my LLM model to Dockerhub, so I don't get haunted by gcp egress cost...

ashley•2/19/24, 11:00 AM

1. Enable FlashBoot if you haven't already done so.
2. Load the model outside of

runpod.serverless.start()

runpod.serverless.start()

so that it is cached into the worker and not loaded on every single request.

TumbleWeedOP•2/19/24, 11:00 AM

So, its possible to preload the model into the worker ?

ashley•2/19/24, 11:00 AM

You can also look at setting Active workers, but you are charged for those.

TTumbleWeed So, its possible to preload the model into the worker ?

ashley•2/19/24, 11:01 AM

Thats basically what FlashBoot does but it doesn't really provide any benefit unless you have a constant flow of requests.

TumbleWeedOP•2/19/24, 11:03 AM

I would try the 2nd option and let you know the result

TTumbleWeed So, its possible to preload the model into the worker ?

J.•2/19/24, 7:02 PM

No

But what he is saying is do:

model = load(model)

def handler():
model.predict()

J.•2/19/24, 7:03 PM

the load model will get added to ur delay time

J.•2/19/24, 7:03 PM

but on subsequent requests if the worker is still active and didnt spin down to take other requests

J.•2/19/24, 7:03 PM

doesnt need to reload into memory

J.•2/19/24, 7:03 PM

if u had it in function scope would reset the variable

Alpay Ariyak•2/21/24, 5:47 AM

Hi @WillyRL, is there a reason you don't want to use https://github.com/runpod-workers/worker-vllm ? It solves all of your problems already

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

Run LLM Model on Runpod Serverless

Similar Threads

Similar Threads

Similar Threads