Trying to work with: llama3-70b-8192 and I get out of memory
Hi
I am trying to work with the model:
llama3-70b-8192 but I cant deploy my serverless endpoint because out of memory.
I have attached image config screenshot. please reccoment on other settings to make it work
[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 896.00 MiB. GPU
Thanks...
Consistently timing out after 90 seconds
I'm not exactly sure why this is happening, and I don't think this happened earlier, but currently I'm consistently seeing requests timeout after 90 seconds.
Max. execution time is set to 300 seconds, so this shouldn't be the issue.
Is this a known problem?...
Serverless problems since 10.12
I am using serverless since few months quit stable, but since 10.12 all my requests execute after 25 seconds i already tried all different settings but at the end the process stops after 25 seconds and i get a error. I changed nothing on Docker ore on my files same settings since weeks.
{
"delayTime": 4967,
"error": "Error queuing workflow: <urlopen error [Errno 111] Connection refused>",...
VLLM model loading, TTFT unhappy path
I am looking for a way to reduce the latency for the unhappy path of VLLM endpoints.
I use the quickstart VLLM template, backed by a network storage for model weights and flashboot enabled. By default the worker will load the model weights on first request. This, however poses the risk of exposing my customers to an unhappy path of latency measured in minutes, at scale we could see this in significant absolute numbers.
What would be the best way for me to make sure that a worker is considered ready only >after< it has loaded the model checkpoints, and trigger checkpoint loading without sending the first request? Should I roll my own VLLM container image? Or is there an idiomatic way to parametrize the quickstart template to achieve this? I would prefer to use the Runpod supplied, properly supported VLLM image, if possible....
