Running llama 3.3 70b using vLLM and 160gb network volume
Hi, I want to check if 160 gb is enough for llama 70b and whether I can use use a smaller network volume
NOr if I need a larger network volume
JMaybe try 200gb first,
Nok thanks a lot
JIf im not wrong it's around 170+, need to check how much on the hf files
JAnd which gpu are going to use?
NI'm not sure atm, are the 24vram options fine?
NI think I'm going to use the suggested option (A6000, A40) and use aqm quant
JOh using quantization
NI set up with vllm template without quant for now using a6000,A40, using 210gb of volume in Canada. I posted an inital request. How long will this take to initialize roughly?
N
JCheck the logs, im not sure how long, depending on the gpu setup you are using too
JBecause I've never deployed that model with that specific quantization
NIt's past 20 mins now
Nwith no quantization
NJHow is the log
JAny signs of out of memory?
N5 items
"endpointId":"ikmbyelhctz06j"
"workerId":"2zeadzwvontveg"
"level":"error"
"message":"Uncaught exception | <class 'torch.OutOfMemoryError'>; CUDA out of memory. Tried to allocate 896.00 MiB. GPU 0 has a total capacity of 44.45 GiB of which 444.62 MiB is free. Process 1865701 has 44.01 GiB memory in use. Of the allocated memory 43.71 GiB is allocated by PyTorch, and 1.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables); <traceback object at 0x7f0a94eff580>;"
"dt":"2024-12-11 05:47:39.26656704"
"endpointId":"ikmbyelhctz06j"
"workerId":"2zeadzwvontveg"
"level":"error"
"message":"Uncaught exception | <class 'torch.OutOfMemoryError'>; CUDA out of memory. Tried to allocate 896.00 MiB. GPU 0 has a total capacity of 44.45 GiB of which 444.62 MiB is free. Process 1865701 has 44.01 GiB memory in use. Of the allocated memory 43.71 GiB is allocated by PyTorch, and 1.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables); <traceback object at 0x7f0a94eff580>;"
"dt":"2024-12-11 05:47:39.26656704"
A guide to torch.cuda, a PyTorch module to run CUDA operations
Nyes
Nsomething must be wrong with my setup
JOom
JSo you add gpus
Nadd workers?
Nahh ok! Is 3 enough
JUse more gpu per worker
JIn edit endpoint, not max workers
Nok!
Nthank you!
JYup
JI'll try to Calc the vram requirement after this if I can
JWhat quantization type do you use
Ngonna try awq but I am a noob, gonna do some research
NI'm using 5 gpus per worker, keeps exiting with error code 1
JUse 4
JAnd don't forger to change the tensor pararelism
JMaybe just create a new endpoint from the quick deploy and because once you deploy it, the config settings will be converted into env variables and if you don't change the env won't be there
JWhen creating a new endpoj change the tensor pararelism to 4 or the amount that you use
JMust be multiple of 2 I guess I but cant be 6
Nok I'll try creating a new endpoint
N
Ncant seem to get any responses
Nno errors in the 
Nlogs
Jis it still loading
Jhows the logs now?
Ji also don't think thats how you use quantization, you need to have the quantized model first then use that. not from unquantized model then load it using quantization in vllm
Jhttps://huggingface.co/ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4
maybe check this out, i found it just now, im not sure if its safe but seems to be a quantized version for it
maybe check this out, i found it just now, im not sure if its safe but seems to be a quantized version for it
J4096 max len
