Runpod•2y ago•

50 replies

Guide to deploy Llama 405B on Serverless?

Hi, can any experts on Serverless advice on how to deploy Llama 405B on Serverless?

Suba•7/25/24, 9:36 AM

@octopus - you need to attach a network volume to the end point. The volume should have at least 1 TB space to hold the 405 B model (unless you are using quantized models). Then increase the number of workers to match the model gpu requirement (like 10 48 GB GPUs)

Suba•7/25/24, 9:37 AM

I tried several 405 B models in HF but get error related to rope_scaling. Looks like we need to modify it to null and try. To do this I need to download all files and upload again.

Jason•7/26/24, 4:25 AM

does the vllm worker supports this yet?

Suba•7/26/24, 6:10 AM

@nerdylive not sure about this, do we have a document or page that lists vllm's support for a model?

Jason•7/26/24, 6:11 AM

on the docs of vllm, not on runpod

Jason•7/26/24, 6:11 AM

look at the right versions, maybe current vllm is outdated

Suba•7/26/24, 6:11 AM

https://docs.vllm.ai/en/latest/models/supported_models.html

Jason•7/26/24, 6:11 AM

yes that one is for the latest version

Jason•7/26/24, 6:11 AM

hope vllm-worker now is the latest

Suba•7/26/24, 6:11 AM

looks like it supports

Suba•7/26/24, 6:12 AM

LlamaForCausalLM

Llama 3.1, Llama 3, Llama 2, LLaMA, Yi

meta-llama/Meta-Llama-3.1-405B-Instruct, meta-llama/Meta-Llama-3.1-70B, meta-llama/Meta-Llama-3-70B-Instruct, meta-llama/Llama-2-70b-hf, 01-ai/Yi-34B, etc.

JJason look at the right versions, maybe current vllm is outdated

Jason•7/26/24, 6:12 AM

okay, again..

Jason•7/26/24, 6:12 AM

check the current vllm worker's vllm version

Jason•7/26/24, 6:12 AM

i think last time it hasn't been updated yet

Suba•7/26/24, 6:13 AM

I am using runpod/worker-vllm:stable-cuda12.1.0

Suba•7/26/24, 6:13 AM

since I am using serverless I am unable to run any command

Jason•7/26/24, 6:14 AM

and does it working with llama3.1 now?

Jason•7/26/24, 6:14 AM

yeah ofc, leme check the repo one sec

Suba•7/26/24, 6:14 AM

No I get error related to rope_scaling

Suba•7/26/24, 6:14 AM

llama 3.1 's config.json has lots of params under rope_scaling

Jason•7/26/24, 6:14 AM

rope scaling huh, i think you're unable to set that too for current vllm worker version

Suba•7/26/24, 6:15 AM

but the current vllm accepts only two params

Jason•7/26/24, 6:15 AM

this is current's vllm-worker docs:
https://docs.vllm.ai/en/v0.3.2/models/supported_models.html

Suba•7/26/24, 6:15 AM

2024-07-24T04:42:22.063990694Z engine.py :110 2024-07-24 04:42:22,063 Error initializing vLLM engine: rope_scalingrope_scaling must be a dictionary with two fields, typetype and factorfactor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

JJason this is current's vllm-worker docs: https://docs.vllm.ai/en/v0.3.2/models/suppor...

Jason•7/26/24, 6:15 AM

yep, check this

Jason•7/26/24, 6:15 AM

thats the matching docs for current vllm-worker

Suba•7/26/24, 6:16 AM

ok got it, 405 is not in there

Jason•7/26/24, 6:16 AM

seems like it just got updated on the newest version yeah

Jason•7/26/24, 6:16 AM

soo only the newest version, and we have to wait until vllm-worker updates to the latest or stable version of vllm

Suba•7/26/24, 6:17 AM

ok.. is it done automatically or should we raise a ticket etc

Jason•7/26/24, 6:18 AM

yeah. about that, we just wait until runpod's staff updates it

Jason•7/26/24, 6:18 AM

they say they're working on it, don't worry

Jason•7/26/24, 6:18 AM

im also waiting for it

Suba•7/26/24, 6:18 AM

great, thank you very much for your time

Suba•7/26/24, 6:18 AM

tim•7/26/24, 8:27 AM

You could try to use https://docs.runpod.io/tutorials/serverless/cpu/run-ollama-inference, but with a GPU. The ollama worker was updated and now it supports also Llama 3.1. We only tested this with 8B, but I don’t see why this shouldn’t also work with 405B

Jason•7/26/24, 8:28 AM

ah ollama interesting

Jason•7/26/24, 8:28 AM

thanks for sharing it will look at that too hahah

tim•7/26/24, 8:36 AM

I will also test this later today with 70 and 405.

Suba•7/31/24, 1:38 PM

@nerdylive would like to know if you got any news on the vllm update

Suba•7/31/24, 1:38 PM

for 405

Jason•7/31/24, 1:45 PM

No not yet I don't know

Jason•7/31/24, 1:45 PM

They're still working on it

Suba•7/31/24, 2:15 PM

@NERDDISCO pls let me know if ollama worker worked with 405

SSuba @NERDDISCO pls let me know if ollama worker worked with 405

yhlong00000•7/31/24, 6:29 PM

https://blog.runpod.io/run-llama-3-1-405b-with-ollama-a-step-by-step-guide/

RunPod Blog

Run Llama 3.1 405B with Ollama: A Step-by-Step Guide

Meta’s recent release of the Llama 3.1 405B model has made waves in the AI community. This groundbreaking open-source model not only matches but even surpasses the performance of leading closed-source models. With impressive scores on reasoning tasks (96.9 on ARC Challenge and 96.8 on GSM8K)