Issue loading a heavy-ish (HuggingFaceM4/idefics2-8b) model on serverless (slow network?)
Hey there,
I'm trying to load the https://huggingface.co/HuggingFaceM4/idefics2-8b model into a serverless worker but i'm running into an issue.
I'm loading the model outside the
When starting the instances, the model starts downloading from HF, but this takes a awfully long time. So long indeed that the serverless handler seems to never start. And the process starts again on another instance, in loop. My guess is that after X amount of time if the worker doesn't expose the handler function, runpod kills it.
The thing is, the model is "only" 35Gb in size. Loading the model on my laptop using my home bandwidth takes only a few minutes.
It seems then that the bandwidth allocation for serverless workers is too limited? I feel like this has changed in the past couple of weeks, I never had issues with this in the past.
Am I missing something here?
I'm trying to load the https://huggingface.co/HuggingFaceM4/idefics2-8b model into a serverless worker but i'm running into an issue.
I'm loading the model outside the
handlerWhen starting the instances, the model starts downloading from HF, but this takes a awfully long time. So long indeed that the serverless handler seems to never start. And the process starts again on another instance, in loop. My guess is that after X amount of time if the worker doesn't expose the handler function, runpod kills it.
The thing is, the model is "only" 35Gb in size. Loading the model on my laptop using my home bandwidth takes only a few minutes.
It seems then that the bandwidth allocation for serverless workers is too limited? I feel like this has changed in the past couple of weeks, I never had issues with this in the past.
Am I missing something here?
