Download Stuck - Worker VLLM

R

reavatarOP•12/18/23, 2:07 PM

This has started happening with new worker vllm changes and wasn't happening earlier

J

Justin Merrell•12/18/23, 3:44 PM

What is the full docker build command you are using? @propback

JJustin Merrell What is the full docker build command you are using? @propback

R

reavatarOP•12/18/23, 5:03 PM

sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .

sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .

sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .

sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .

R

reavatarOP•12/18/23, 5:04 PM

With username/image:tag ofcourse my own

R

reavatarOP•12/18/23, 5:04 PM

This is same command listed on Readme docs

A

Alpay Ariyak•12/18/23, 9:12 PM

Hey, how long did you keep it running?

AAlpay Ariyak Hey, how long did you keep it running?

R

reavatarOP•12/19/23, 8:05 AM

40 minutes

AAlpay Ariyak Hey, how long did you keep it running?

R

reavatarOP•12/19/23, 11:38 AM

I previously had cloned till below commit and was working.
https://github.com/runpod-workers/worker-vllm/tree/24feadd903cf5528b3e5ec8ce400fd6a184ecc04
Now, when using this commit

# Quick temporary updates
RUN pip install git+https://github.com/runpod/runpod-python@a1#egg=runpod --compile

# Quick temporary updates
RUN pip install git+https://github.com/runpod/runpod-python@a1#egg=runpod --compile

# Quick temporary updates
RUN pip install git+https://github.com/runpod/runpod-python@a1#egg=runpod --compile

# Quick temporary updates
RUN pip install git+https://github.com/runpod/runpod-python@a1#egg=runpod --compile

is causing trouble. If this could be worked out, I'd atleast having something working for now

A

Alpay Ariyak•12/20/23, 6:30 PM

Could you try now please?

A

Alpay Ariyak•12/20/23, 11:21 PM

Fixed in latest version. The only thing you can't do atm is build from a machine without GPUs

AAlpay Ariyak Fixed in latest version. The only thing you can't do atm is build from a machine...

R

reavatarOP•12/21/23, 5:17 AM

It crash my 32GB RAM while running setup.py vllm

R

reavatarOP•12/21/23, 5:19 AM

Also tried using

runpod/worker-vllm:dev

runpod/worker-vllm:dev

runpod/worker-vllm:dev

runpod/worker-vllm:dev directly but while running container on runpod

R

reavatarOP•12/21/23, 5:20 AM

Also, thought to build docker image on runpod gpu, but docker wasn't starting there

Rreavatar Also tried using `runpod/worker-vllm:dev` directly but while running container o...

A

Alpay Ariyak•12/21/23, 5:20 AM

@merrell.io I think you need to update the GitHub action for pushing the image, it still hasn’t from my last commit which was 4h ago

Rreavatar Also, thought to build docker image on runpod gpu, but docker wasn't starting th...

A

Alpay Ariyak•12/21/23, 5:21 AM

RunPod machines are inside docker containers, so you can’t run docker inside

A

Alpay Ariyak•12/21/23, 5:24 AM

You can use this in the meantime alpayariyak/vllm:11.8 @reavatar

AAlpay Ariyak You can use this in the meantime alpayariyak/vllm:11.8 @reavatar

R

reavatarOP•12/21/23, 5:27 AM

error pulling image: Error response from daemon: manifest for alpayariyak/vllm:11.8 not found: manifest unknown: manifest unknown

error pulling image: Error response from daemon: manifest for alpayariyak/vllm:11.8 not found: manifest unknown: manifest unknown

error pulling image: Error response from daemon: manifest for alpayariyak/vllm:11.8 not found: manifest unknown: manifest unknown

error pulling image: Error response from daemon: manifest for alpayariyak/vllm:11.8 not found: manifest unknown: manifest unknown

A

Alpay Ariyak•12/21/23, 5:27 AM

Try again

AAlpay Ariyak Try again

R

reavatarOP•12/21/23, 5:28 AM

ok, downloading now

AAlpay Ariyak Try again

R

reavatarOP•12/21/23, 5:36 AM

worked. Thanks

A

Alpay Ariyak•12/21/23, 5:36 AM

No problem!

AAlpay Ariyak No problem!

R

reavatarOP•12/22/23, 8:23 AM

By the way, this doesn't return logprobs even when passed in sampling params

Rreavatar By the way, this doesn't return logprobs even when passed in sampling params

A

Alpay Ariyak•12/22/23, 8:24 AM

Yeah, it's not supported atm

AAlpay Ariyak Yeah, it's not supported atm

R

reavatarOP•12/22/23, 8:28 AM

Any ETA?

Rreavatar Any ETA?

A

Alpay Ariyak•12/22/23, 8:33 AM

Anything between 1 and 3 weeks really, it's a bit in the backlog, as there's very little demand for it. I'm curious, what do you need them for?

AAlpay Ariyak Anything between 1 and 3 weeks really, it's a bit in the backlog, as there's ver...

R

reavatarOP•12/22/23, 8:38 AM

Using it create labels eg. True/False and want to pick highest prob labels and also have a kind of system where I need to verify if a particular response was generated from a model or not. I think I'd just switch to Modal for now.

Download Stuck - Worker VLLM

Similar Threads

Similar Threads

Similar Threads