Custom Template Taking Hours To Initialize

Ryan Witzman · 2024-01-03T21:47:35.873Z

I made a custom template with the docker template https://huggingface.co/spaces/rwitz/go-bruins-v2 and it is taking hours to initialize on serverless.

J

J.•1/3/24, 9:48 PM

Wait I am confused what you are trying to do?

You are trying to initialize this on serverless? or is this a docker building?

J

J.•1/3/24, 9:49 PM

Im just confused cause this by itself already has issues

J

J.•1/3/24, 9:49 PM

@Ryan Witzman (rwitz) What is your min / max workers

J

J.•1/3/24, 9:49 PM

there should be workers

J

J.•1/3/24, 9:49 PM

do u have:
1) your runpod template that you used to start your serverless
2) what is this?

J

J.•1/3/24, 9:49 PM

3) do you have your docker file

R

Ryan WitzmanOP•1/3/24, 9:50 PM

min workers is 0 and max is 1

JJ.do u have: 1) your runpod template that you used to start your serverless 2) wha...

R

Ryan WitzmanOP•1/3/24, 9:50 PM

yeah its on huggingface

JJ.3) do you have your docker file

R

Ryan WitzmanOP•1/3/24, 9:50 PM

yes

R

Ryan WitzmanOP•1/3/24, 9:50 PM

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

CMD [ "python", "-u", "/handler.py" ]

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

CMD [ "python", "-u", "/handler.py" ]

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

CMD [ "python", "-u", "/handler.py" ]

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

CMD [ "python", "-u", "/handler.py" ]

J

J.•1/3/24, 9:50 PM

Can I see your docker file?

And also... Im not sure if hugging face holds docker images?

J

J.•1/3/24, 9:50 PM

I could be wrong

J

J.•1/3/24, 9:50 PM

For #1, there is something on runpod you used to create it

J

J.•1/3/24, 9:51 PM

J

J.•1/3/24, 9:51 PM

what i mean is this

R

Ryan WitzmanOP•1/3/24, 9:51 PM

oh

R

Ryan WitzmanOP•1/3/24, 9:51 PM

J

J.•1/3/24, 9:51 PM

Ah interesting

J

J.•1/3/24, 9:51 PM

I learnt something new

J

J.•1/3/24, 9:52 PM

Okay, so just as an FYI:
I recommend trying to build a template using:
FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

And don't do the run command, and test it using GPU Pod

it always is a great place to test.

2) https://huggingface.co/spaces/rwitz/go-bruins-v2

Your build failed

J

J.•1/3/24, 9:52 PM

So im guessing that your serverless function

J

J.•1/3/24, 9:52 PM

is trying to find something

J

J.•1/3/24, 9:52 PM

that doesn't exist

R

Ryan WitzmanOP•1/3/24, 9:52 PM

yeah because the huggingface doesnt have cuda

R

Ryan WitzmanOP•1/3/24, 9:53 PM

the serverless should have cuda

J

J.•1/3/24, 9:53 PM

Serverless does have cuda, but if your docker build failed, it won't produce an image for serverless to use

J

J.•1/3/24, 9:53 PM

that means in your docker build step, you gotta maybe default to using CPU to load the model

J

J.•1/3/24, 9:53 PM

vs a torch cuda

J

J.•1/3/24, 9:53 PM

so that the image builds successfully

J

J.•1/3/24, 9:54 PM

do you see the :latest tag somewhere tho?

J

J.•1/3/24, 9:54 PM

just curious, never used hugging face myself

J

J.•1/3/24, 9:54 PM

to host an image registry

J

J.•1/3/24, 9:55 PM

What is happening is when you run the build command:
1. It sets up an isolated repeatable environment
2. If it fails to set up that isolated environment it won't produce an "image" for other environment to start off of

So #2 is what is happening.

3. Gotta download the models without using like torch.cuda etc.

R

Ryan WitzmanOP•1/3/24, 9:56 PM

Ok so I will have the docker download the model instead of instantiated it

R

Ryan WitzmanOP•1/3/24, 9:56 PM

yes

J

J.•1/3/24, 9:56 PM

can prob just use a curl request or somethign to the models directly

J

J.•1/3/24, 9:57 PM

Also, ive never used hugging face as a docker registry before, but just make it public? i guess haha if not already. im guessing there might be issues if its a private repository, but ive never used it before

J

J.•1/3/24, 9:57 PM

i usually just use dockerhub

J

J.•1/3/24, 9:57 PM

gl gl!

JJ.Also, ive never used hugging face as a docker registry before, but just make it ...

R

Ryan WitzmanOP•1/3/24, 9:57 PM

yes

J

J.•1/3/24, 9:57 PM

I also recommend again:

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel

RUN pip install runpod transformers

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

RUN python -c 'from transformers import pipeline; import torch; pipe = pipeline("text-generation", model="rwitz/go-bruins-v2",device=0,torch_dtype=torch.bfloat16)'

ADD handler.py .

if you end up wanting to test it on a GPU Pod, vs serverless, i always find that an easier validation step

J

J.•1/3/24, 9:57 PM

then in your serverless, you can do as you did, where you overrode the CMD command

J

J.•1/3/24, 9:58 PM

or you can have a second docker file that does:

FROM MYGPU POD IMAGE

ADD handler.py .

CMD handler.py all that

FROM MYGPU POD IMAGE

ADD handler.py .

CMD handler.py all that

FROM MYGPU POD IMAGE

ADD handler.py .

CMD handler.py all that

FROM MYGPU POD IMAGE

ADD handler.py .

CMD handler.py all that

J

J.•1/3/24, 9:58 PM

Which would make your future iterations of handler.py super fast

J

J.•1/3/24, 9:58 PM

Since all you got is now a base image with all the models - not needing to rebuild them every time, and all u gotta do is add your new handler.py and stuff

JJ.Since all you got is now a base image with all the models - not needing to rebui...

R

Ryan WitzmanOP•1/3/24, 9:59 PM

ah i see

J

J.•1/3/24, 10:00 PM

Yup yup~ is a new thing i learnt last week haha, cause i have an audio sound effect stuff, that I was playing with some big models

J

J.•1/3/24, 10:00 PM

and i was getting painful to keep downloading the model

J

J.•1/3/24, 10:00 PM

on every iteration haha

Custom Template Taking Hours To Initialize

Similar Threads

Similar Threads

Similar Threads