Delay Time

Minozar · 2024-05-17T19:45:11.070Z

Hello, I'm wondering if those Delay Time are normal? If not, what should I do?

D

digigoblin•5/17/24, 8:15 PM

Seems pretty low to me. Depends on what your worker does.

M

MinozarOP•5/25/24, 11:34 AM

It takes a bunch of image urls and do ML inference on them

M

MinozarOP•5/25/24, 11:34 AM

Maybe one question, does the delay time increase if the "active and supposed to be warm" worker hasn't actually been working for a while?

M

MinozarOP•5/25/24, 11:38 AM

it kinda feels unreliable atm

M

MinozarOP•5/25/24, 11:48 AM

M

MinozarOP•5/25/24, 11:49 AM

@Papa Madiator

M

Madiator2011•5/25/24, 11:49 AM

?

M

MinozarOP•5/25/24, 11:53 AM

I'm experiencing extreme delay time, how can I get them back to normal?

M

Madiator2011•5/25/24, 12:07 PM

I'm not sure what are you doing and running

M

MinozarOP•5/25/24, 12:17 PM

atm it just takes a bunch of image urls and calculate their embeddings using a specific ML model

M

Madiator2011•5/25/24, 12:18 PM

did you bake models into docker image?

M

MinozarOP•5/25/24, 12:23 PM

my dockerfile looks like this:

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu20.04

RUN apt-get update && apt-get install -y \
    python3-pip \
    wget \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN pip install runpod==1.6.2 torch torchvision torchaudio sentence-transformers==2.7.0

# caching the model
RUN python3 -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('clip-ViT-B-32', device='cpu')"

COPY builder/requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY src/ .

CMD ["python3", "handler.py"]

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu20.04

RUN apt-get update && apt-get install -y \
    python3-pip \
    wget \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN pip install runpod==1.6.2 torch torchvision torchaudio sentence-transformers==2.7.0

# caching the model
RUN python3 -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('clip-ViT-B-32', device='cpu')"

COPY builder/requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY src/ .

CMD ["python3", "handler.py"]

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu20.04

RUN apt-get update && apt-get install -y \
    python3-pip \
    wget \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN pip install runpod==1.6.2 torch torchvision torchaudio sentence-transformers==2.7.0

# caching the model
RUN python3 -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('clip-ViT-B-32', device='cpu')"

COPY builder/requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY src/ .

CMD ["python3", "handler.py"]

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu20.04

RUN apt-get update && apt-get install -y \
    python3-pip \
    wget \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN pip install runpod==1.6.2 torch torchvision torchaudio sentence-transformers==2.7.0

# caching the model
RUN python3 -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('clip-ViT-B-32', device='cpu')"

COPY builder/requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY src/ .

CMD ["python3", "handler.py"]

MMinozar my dockerfile looks like this: ```dockerfile FROM nvidia/cuda:12.4.1-cudnn-ru...

J

Jason•5/25/24, 12:57 PM

Woah the python line im not sure its caching it in runtime

J

Jason•5/25/24, 12:57 PM

run it on CMDor ENTRYPOINTfile

J

Jason•5/25/24, 12:57 PM

not on the build but on the runtime

M

MinozarOP•5/25/24, 12:59 PM

This python line does two things:
- it downloads the model and save it somewhere
- it loads it into the ram

If I remove the line and only do that on run time, I'd have to download it each time no ?

M

MinozarOP•5/25/24, 12:59 PM

(or maybe I misunderstood something sorry for that)

J

Jason•5/25/24, 12:59 PM

Do it on the runtime too

J

Jason•5/25/24, 1:00 PM

its fine

MMinozar This python line does two things: - it downloads the model and save it somewhere...

J

Jason•5/25/24, 1:00 PM

Yess but it loads on your build machine not on the runtime

J

Jason•5/25/24, 1:00 PM

then on the runtime its gonna be different machine so its not cached yet

MMadiator2011 did you bake models into docker image?

M

MinozarOP•5/25/24, 1:01 PM

Usually whats the proper way to bake models into docker images?

Any examples?

J

Jason•5/25/24, 1:01 PM

Like on the docs

J

Jason•5/25/24, 1:01 PM

It loads but on the runtime using a python file / .sh file

M

Madiator2011•5/25/24, 1:01 PM

I would check logs when worker is starting

J

Jason•5/25/24, 1:03 PM

If your handler requires external files such as model weights, be sure to cache them into your docker image. You are striving for a completely self-contained worker that doesn't need to download or fetch external files to run.

Oh on the docs its only like that

J

Jason•5/25/24, 1:04 PM

I guess running the python line that loads the model or creates the pipeline works in handler.py works

JJason 🚧 If your handler requires external files such as model weights, be sure to cac...

M

MinozarOP•5/25/24, 4:44 PM

exactly, so that's not very clear to me how to cache models in the docker image, I thought my implementation would work

JJason I guess running the python line that loads the model or creates the pipeline wor...

M

MinozarOP•5/25/24, 4:46 PM

import numpy as np

# Can I load the model here??

def handler(job):
    return True

runpod.serverless.start({"handler": handler})

import numpy as np

# Can I load the model here??

def handler(job):
    return True

runpod.serverless.start({"handler": handler})

import numpy as np

# Can I load the model here??

def handler(job):
    return True

runpod.serverless.start({"handler": handler})

import numpy as np

# Can I load the model here??

def handler(job):
    return True

runpod.serverless.start({"handler": handler})

Is that what you meant?

J

Jason•5/25/24, 4:46 PM

Yep

M

MinozarOP•5/25/24, 4:47 PM

So everything put outside of the handler function will be cached in the docker image??

J

Jason•5/25/24, 4:47 PM

before the start i think

J

Jason•5/25/24, 4:47 PM

serverless.start

JJason 🚧 If your handler requires external files such as model weights, be sure to cac...

M

MinozarOP•5/25/24, 4:49 PM

Yes but the documentation says "be sure to cache them into your docker image."
How to do that correctly? (the doc doesn't provide enough information I think)

J

Jason•5/25/24, 4:51 PM

Just before the start() line

J

Jason•5/25/24, 4:51 PM

so it has to be loaded before that i think

M

MinozarOP•5/25/24, 4:53 PM

If I have no active worker, each time I'll spawn a new one, It'll dowload the model, that's not what I want, I'd like the model to be cached in the docker image so when I spawn a new worker, the model is already almost ready to use

J

Jason•5/25/24, 4:54 PM

cached means it has to be stored somewhere

M

Madiator2011•5/25/24, 4:55 PM

also good idea is that in handler you load model to vram

J

Jason•5/25/24, 4:55 PM

Either you store it in the image or network storage first

J

Jason•5/25/24, 4:55 PM

then load it into the vram in runtime thats how you cache it

J

Jason•5/25/24, 4:55 PM

so if you want to download the model, download it into network volume

MMinozar ```py import numpy as np # Can I load the model here?? def handler(job): r...

J

Jason•5/25/24, 4:56 PM

@Papa Madiator so like this is fine right?

M

Madiator2011•5/25/24, 4:56 PM

yup

M

Madiator2011•5/25/24, 4:57 PM

examples https://github.com/runpod-workers/worker-sdxl/blob/main/src/rp_handler.py

GitHub

worker-sdxl/src/rp_handler.py at main · runpod-workers/worker-sdxl

RunPod worker for Stable Diffusion XL. Contribute to runpod-workers/worker-sdxl development by creating an account on GitHub.

JJason @Papa Madiator so like this is fine right?

M

MinozarOP•5/25/24, 5:05 PM

but this is only to load the model in VRAM, first I'd need to download it from somewhere
after reading the documentation I understand it has to be in from the cache of the image, so it has to be linked with the dockerfile somehow

MMadiator2011 examples https://github.com/runpod-workers/worker-sdxl/blob/main/src/rp_handler....

M

MinozarOP•5/25/24, 5:06 PM

ok update, I just checked the dockerfile of the project you shared

M

MinozarOP•5/25/24, 5:06 PM

# Base image
FROM runpod/base:0.4.2-cuda11.8.0

ENV HF_HUB_ENABLE_HF_TRANSFER=0

# Install Python dependencies (Worker Template)
COPY builder/requirements.txt /requirements.txt
RUN python3.11 -m pip install --upgrade pip && \
    python3.11 -m pip install --upgrade -r /requirements.txt --no-cache-dir && \
    rm /requirements.txt

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Add src files (Worker Template)
ADD src .

CMD python3.11 -u /rp_handler.py

# Base image
FROM runpod/base:0.4.2-cuda11.8.0

ENV HF_HUB_ENABLE_HF_TRANSFER=0

# Install Python dependencies (Worker Template)
COPY builder/requirements.txt /requirements.txt
RUN python3.11 -m pip install --upgrade pip && \
    python3.11 -m pip install --upgrade -r /requirements.txt --no-cache-dir && \
    rm /requirements.txt

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Add src files (Worker Template)
ADD src .

CMD python3.11 -u /rp_handler.py

# Base image
FROM runpod/base:0.4.2-cuda11.8.0

ENV HF_HUB_ENABLE_HF_TRANSFER=0

# Install Python dependencies (Worker Template)
COPY builder/requirements.txt /requirements.txt
RUN python3.11 -m pip install --upgrade pip && \
    python3.11 -m pip install --upgrade -r /requirements.txt --no-cache-dir && \
    rm /requirements.txt

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Add src files (Worker Template)
ADD src .

CMD python3.11 -u /rp_handler.py

# Base image
FROM runpod/base:0.4.2-cuda11.8.0

ENV HF_HUB_ENABLE_HF_TRANSFER=0

# Install Python dependencies (Worker Template)
COPY builder/requirements.txt /requirements.txt
RUN python3.11 -m pip install --upgrade pip && \
    python3.11 -m pip install --upgrade -r /requirements.txt --no-cache-dir && \
    rm /requirements.txt

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Add src files (Worker Template)
ADD src .

CMD python3.11 -u /rp_handler.py

M

MinozarOP•5/25/24, 5:07 PM

-------------
I think this is what I was looking for

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

# Cache Models
COPY builder/cache_models.py /cache_models.py
RUN python3.11 /cache_models.py && \
    rm /cache_models.py

MMinozar ------------- I think this is what I was looking for ```dockerfile # Cache Model...

J

Jason•5/25/24, 5:08 PM

Nono dont use RUN.. use CMD or Entrypoint

Similar Threads

Similar Threads

Similar Threads