Run Mixtral 8x22B Instruct on vLLM worker
Hello everybody, is it possible to run mixtral 8x22B on vLLM worker i tried to run it on the default configuration with 48 gb GPU A6000, A40 but its taking too long, what are the requirements for running mixtral 8x22B successfully ? this is the model that im trying to run https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1
77 Replies
Sorry im new to using GPU's for LLM models
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
thanks for the reply what do you mean by half
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
let me check which GPU would be suitable to run this btw ?
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
sure
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
i think this would also be a good option to set right ? since it will divide the memory
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
ah i see, would there be any substantial decrease in quality if i ran the model in half memory ?
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
cool thanks
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
yeah makes sense
@nerdylive looks like mixtral 8x2b requires up to 300 gb of vram, and the highest available gpu is 80gb of vram if it uses half amount of memory which would be 150 gb it should be possible to divide 50gb of vram between 3 workers. idk if thats possible do you know somebody from the team that can help me out here ? my company actually wants to deploy this model for our product.
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
from the mistral discord
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
i see let me search, though could somebody from the runpod team confirm this ?
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
ah ok, i misunderstood this is a community server
sorry
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
Yeah i will contact through official channels thanks for all the help appreciate it
how do i mark this post as solved ?
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
the only way i have right now is to use a vm with 300gb of vram but it would be costly and not sure if i can find a vm like that, i opted for runpod because it had cheap pricing and easy deployments
sure i will post updates here
one guy in mistral discord also wanted to split memory in order to run the model between 4x gpus
they suggested vLLM for this which is what runpod workers are using i think
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
i havent looked into it yet but they suggested it and TGI
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
yeah
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
@nerdylive this is what i got
https://docs.mistral.ai/deployment/self-deployment/vllm/ in this guide they set the tensor parallel size to 4 i wonder if runpod does it as well
vLLM | Mistral AI Large Language Models
vLLM can be deployed using a docker image we provide, or directly from the python package.
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
let me check
oh this was the option lol
i set it to 3 but still ran out of memory
i used 2 gpu per worker as well actually 80gb
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
no i ran out of memory
even with the above config
I'm trying to do the same thing as you right now lol will update if I figure something out
Thanks alot
@Alpay Ariyak mayby you could help here π
Hi, you need at least 2x80Gb GPUs afaik
Hey, yes I used 2x 80 GB GPU per worker with 3 workers but I got an error
torch.cuda ran out of memory while trying to allocateUnknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
yeah i will try soon
i just selected the option 2 gpu per worker and 80gb H100
Oh? I can only do 2 GPUs per worker with 48GB GPUs, not 80GB GPUs. Are you sure?
Unless you're doing a pod instead of serverless
In which case ignore me π
My apologies, you actually need 4x80gb for 8x22B
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
Not with the current limits, no
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
With OpenAI compatibility?
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
Not sse, regular get request
Will return yielded outputs from the worker since last /stream call
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
What goal do you have in mind?
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
OpenAI compatibility streaming is through SSE
Hey, sorry to hijack the thread, I'm also looking into deploying vLLM on RunPod serverless. The landing page indicates that it should be possible to bring your own container, not pay for any idle time, and have <250ms cold boot. Is this true? It sounds too good to be true.
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
Yes, through flash boot
That one is strictly polled
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
Does this 250ms cold boot time really include everything? Or does it only contain some things, such that the actual cold boot time might be 30 seconds or something? For example, the time to load LLM weights into memory typically takes more than 10 seconds.
Everything, due to not needing to reload weights
That's just insane if it really works
Haha try it out!
Yeah, reading the docs right now to figure out what is everything i need to do to try it... I currently have a Docker image that spins up a fork of oobabooga web ui, I'm thinking about setting that up for the serverless experiment.
Yeah your actually right, i confused it as 80gb
my bad guys
even with using dtype half ? we need 4x80 gb ?
8x22B = 176B parameters. At 16bit, 2 bytes per parameter, that's 352GB just for the model parameters
At 8bit (1 byte per parameter) it's still 176GB
I could be mistaken around this, I'm not an expert on this for sure
But my understanding is that you can just fit 8x22B on 4x80GB with 8bit quantization
I see yeah that makes sense
i will revisit this in the future
316gb for 16bit according to https://huggingface.co/spaces/Vokturz/can-it-run-llm
Weβre raising the serverless gpu count limits around next week
I believe even up to 10x A40 per worker
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
Yes, 2x of everything at the very least iirc
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View
nice this will be useful thanks alot
Unknown Userβ’2y ago
Message Not Public
Sign In & Join Server To View