Runpod•2y ago•

56 replies

worker-vllm cannot download private model

I built my model successfully and it was able to download it during the build. However, when I deploy it on RunPod Serverless, it fails to startup upon request because it cannot download the model.

export DOCKER_BUILDKIT=1
export HF_TOKEN="your_token"

docker build -t user/app:0.0.1 \
    --secret id=HF_TOKEN \
    --build-arg MODEL_NAME="my_model_path" \
    ./worker-vllm

export DOCKER_BUILDKIT=1
export HF_TOKEN="your_token"

docker build -t user/app:0.0.1 \
    --secret id=HF_TOKEN \
    --build-arg MODEL_NAME="my_model_path" \
    ./worker-vllm

export DOCKER_BUILDKIT=1
export HF_TOKEN="your_token"

docker build -t user/app:0.0.1 \
    --secret id=HF_TOKEN \
    --build-arg MODEL_NAME="my_model_path" \
    ./worker-vllm

export DOCKER_BUILDKIT=1
export HF_TOKEN="your_token"

docker build -t user/app:0.0.1 \
    --secret id=HF_TOKEN \
    --build-arg MODEL_NAME="my_model_path" \
    ./worker-vllm

ashley•1/30/24, 9:28 PM

@Alpay Ariyak any idea?

Alpay Ariyak•1/30/24, 9:30 PM

Try also specifying the model name as an environment variable within the endpoint template

AAlpay Ariyak Try also specifying the model name as an environment variable within the endpoin...

Casper.OP•1/30/24, 9:33 PM

I just followed the guide on the worker-vllm repository

Casper.OP•1/30/24, 9:33 PM

Shouldn't this work?

Casper.OP•1/30/24, 9:34 PM

One of the serverless nodes printed this which seems correct (removed the model)

{
   "model":"<my_model_path>",
   "download_dir":"/runpod-volume",
   "quantization":"None",
   "load_format":"auto",
   "dtype":"auto",
   "disable_log_stats":true,
   "disable_log_requests":true,
   "trust_remote_code":false,
   "gpu_memory_utilization":0.95,
   "max_parallel_loading_workers":48,
   "max_model_len":"None",

{
   "model":"<my_model_path>",
   "download_dir":"/runpod-volume",
   "quantization":"None",
   "load_format":"auto",
   "dtype":"auto",
   "disable_log_stats":true,
   "disable_log_requests":true,
   "trust_remote_code":false,
   "gpu_memory_utilization":0.95,
   "max_parallel_loading_workers":48,
   "max_model_len":"None",

Casper.OP•1/30/24, 9:34 PM

But somehow, it is still not able to find the model

Alpay Ariyak•1/30/24, 9:58 PM

Could you share the error message you get please @casper_ai

Alpay Ariyak•1/31/24, 12:50 AM

I think I got it, will push a fix soon

Alpay Ariyak•1/31/24, 3:14 AM

Hi @Casper. , just pushed the update to main

AAlpay Ariyak Hi @Casper. , just pushed the update to main

Casper.OP•1/31/24, 7:06 AM

Thanks for making the update! I will test later today

CCasper.Thanks for making the update! I will test later today

Alpay Ariyak•1/31/24, 7:15 AM

Of course! Pushed custom jinja chat templates as well

Casper.OP•1/31/24, 3:27 PM

@Alpay Ariyak I'm still getting the same error, although I see the path has changed in the vLLM config

message.txt5.71KB

CCasper.@Alpay Ariyak I'm still getting the same error, although I see the path has chan...

Alpay Ariyak•1/31/24, 3:55 PM

It seems like the issue is that your model doesn’t have a tokenizer

AAlpay Ariyak It seems like the issue is that your model doesn’t have a tokenizer

Casper.OP•1/31/24, 4:45 PM

Casper.OP•1/31/24, 4:45 PM

That can't be right, the tokenizer is there

Casper.OP•1/31/24, 4:45 PM

It says 401 unauthorized

Casper.OP•1/31/24, 4:46 PM

So is the issue that you are not downloading the tokenizer into the directory perhaps?

Casper.OP•1/31/24, 4:49 PM

I thought the idea with downloading the model into the image was to 1) reduce startup time, and 2) having a secure environment with no access to your Huggingface token

Casper.OP•1/31/24, 5:01 PM

https://github.com/runpod-workers/worker-vllm/pull/39

GitHub

Download tokenizer upon build by casper-hansen · Pull Request #39 ·...

This downloads the tokenizer when building the worker-vllm image.
This has the following benefit:

You do not have to send any network request to Huggingface during initialization. This means you d...

Casper.OP•1/31/24, 5:01 PM

Please review @Alpay Ariyak

Alpay Ariyak•1/31/24, 5:04 PM

Looks good, have you built an image with it and tested it?

AAlpay Ariyak Looks good, have you built an image with it and tested it?

Casper.OP•1/31/24, 5:04 PM

I tested that it downloads the tokenizer to the directory specified, but have not ran a deployment yet

Casper.OP•1/31/24, 5:05 PM

I can run a deployment to test it

Casper.OP•1/31/24, 5:05 PM

Any tips on how to push images with models inside to Docker Hub faster? Takes like half an hour even though my internet is speedy

Alpay Ariyak•1/31/24, 5:07 PM

I can test it in a bit, no worries

CCasper.Any tips on how to push images with models inside to Docker Hub faster? Takes li...

Alpay Ariyak•1/31/24, 5:09 PM

Unfortunately not
Is there a reason you’re baking the model in vs using the pre-built image? I haven’t seen too much of a difference in load times thus far

AAlpay Ariyak Unfortunately not Is there a reason you’re baking the model in vs using the pre-...

Casper.OP•1/31/24, 5:11 PM

It's just the best way currently. You avoid all outgoing traffic this way

Casper.OP•1/31/24, 5:11 PM

I think a better alternative could be to use the snapshot_download functionality from Huggingface Hub instead

Casper.OP•1/31/24, 5:11 PM

That way, you make sure you download everythign

Casper.OP•1/31/24, 5:19 PM

Tbh I don't think this PR will solve the issue because it also needs the config and everything else

Casper.OP•1/31/24, 5:26 PM

Okay, I replaced the current download with snapshot_download. Building and deploying in a moment

Alpay Ariyak•1/31/24, 7:10 PM

Will get back to you, unavailable for the next 2 hours

Casper.OP•1/31/24, 8:29 PM

I tried using the PR and I think it must be something else too

message.txt4.1KB

Casper.OP•1/31/24, 8:29 PM

I'm not sure why but the engine.pyengine.py is not able to find the tokenizer

Alpay Ariyak•1/31/24, 8:48 PM

Yeah, tokenizer needs to point to the downloaded path, I’ll fix this up shortly

Casper.OP•1/31/24, 8:48 PM

Wait I might have just compiled from the wrong branch

Casper.OP•1/31/24, 8:49 PM

I think I compiled this from the main branch instead of my PR

AAlpay Ariyak Yeah, tokenizer needs to point to the downloaded path, I’ll fix this up shortly

Casper.OP•1/31/24, 8:50 PM

Thanks, it would be great! I'm still testing my PR as the problem is also that worker-vllm currently does not download the tokenizer or any config files, which will inevitably lead to an error when dealing with private repositories

CCasper.Thanks, it would be great! I'm still testing my PR as the problem is also that w...

Alpay Ariyak•1/31/24, 8:52 PM

Of course! Will fix that for sure
Although what would be the issue with just specifying the env variable HF_TOKEN in the endpoint template? It would allow for the tokenizer download, and tokenizers are always tiny, so should be a quick download

AAlpay Ariyak Of course! Will fix that for sure Although what would be the issue with just spe...

Casper.OP•1/31/24, 8:55 PM

I do not want to add a HF_TOKEN to a production environment since it grants access to everything meant to be kept private

Casper.OP•1/31/24, 8:56 PM

And it will also help with reducing delay as much as possible

Alpay Ariyak•1/31/24, 8:56 PM

So I think we should keep the model download the same and separately snapshot download the tokenizer

Alpay Ariyak•1/31/24, 8:57 PM

This is because if we snapshot download the entire repo, it will download all formats of the model (e.g. both .bin and .safetensors), which would bloat the image size

AAlpay Ariyak So I think we should keep the model download the same and separately snapshot do...

Casper.OP•1/31/24, 8:57 PM

It's not only the tokenizer that should cause issues right?

Casper.OP•1/31/24, 8:58 PM

vLLM also needs to load the model config

Casper.OP•1/31/24, 8:58 PM

Otherwise, I do agree that we can try to minimize bloating the image size

Alpay Ariyak•1/31/24, 9:00 PM

Hmm, didn’t notice any issues with lack of model config yet, but could be an issue potentially as I did not test private models
If anything, we can just have a priority list of formats (+ability to specify directly with LOAD_FORMAT env var) and download this way, similarly to vllm, but also allowing json files and whatever else is needed

AAlpay Ariyak Hmm, didn’t notice any issues with lack of model config yet, but could be an iss...

Casper.OP•1/31/24, 9:17 PM

I would suggest you just upload opt-125m or something small to test with

Casper.OP•1/31/24, 9:17 PM

I modified the PR to only download tokenizer and config

    snapshot_download(
        model,
        cache_dir=download_dir,
        allow_patterns=[
            "*token*",
            "config.json",
        ]
    )

    snapshot_download(
        model,
        cache_dir=download_dir,
        allow_patterns=[
            "*token*",
            "config.json",
        ]
    )

Casper.OP•1/31/24, 9:24 PM

I updated the PR and testing again now
https://github.com/runpod-workers/worker-vllm/pull/39

GitHub

Download full repository upon build by casper-hansen · Pull Request...

This downloads the full repository when building the worker-vllm image.
This has the following benefit:

You do not have to send any network request to Huggingface during initialization. This means...

worker-vllm cannot download private model

Similar Threads

Similar Threads

Similar Threads