Speeding up loading of model weights

Hi guys, I have setup my severless docker image to contain all my required model weights. My handler script also loads the weights using the diffusers library's

.from_pretrained

.from_pretrained

with

 local_files_only=true

 local_files_only=true

so we are loading everything locally. I notice that during cold starts, loading those weights still take around 25 seconds till the logs display

--- Starting Serverless Worker |  Version 1.6.2 ---

--- Starting Serverless Worker |  Version 1.6.2 ---

.

Anyone has experience optimising the time needed tp load weights? Could we pre-load it on ram or something (I may be totally off)?

Runpod•2y ago•

6 replies

Hello

Speeding up loading of model weights

Hi guys, I have setup my severless docker image to contain all my required model weights. My handler script also loads the weights using the diffusers library's

.from_pretrained

.from_pretrained

with

 local_files_only=true

 local_files_only=true

so we are loading everything locally. I notice that during cold starts, loading those weights still take around 25 seconds till the logs display

--- Starting Serverless Worker |  Version 1.6.2 ---

--- Starting Serverless Worker |  Version 1.6.2 ---

.

Anyone has experience optimising the time needed tp load weights? Could we pre-load it on ram or something (I may be totally off)?

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

Runpod

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

21,906 Members

Join

Speeding up loading of model weights

Speeding up loading of model weights

Continue the conversation

Runpod

Continue the conversation

Runpod

Similar Threads