vLLM serverless output cutoff
I deployed a serverless vLLM using deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
But when i made a request, output is only 16 tokens (tested many times), I don't change anything from default setting but max_model_length to 32768.
How can i fix that? or did I miss any config?...
Why the serverless downloading instead of "running" when i trigger the runpod id?
My app is having connection because of this, anyone exp the same thing? Im using github integration approach
OSError in vLLM worker; issues when its new update was released
I was using vLLM worker 1.7.0 and everything was working fine till yesterday. Today I am facing issues in all of my endpoints where huggingface models are deployed using the vLLM worker. Runpod logs shows OSError and the model cant be identified.
I then deployed a new endpoint with latest configuration of vLLM worker 1.9 and everything worked in the way it used to. @Justin Merrell
Runpod should let us know its changes atleast, so it does not affect the endpoints in production....
How much does it cost to use multi-GPU ?
I'd like to increase the number of GPUs per worker to get better performance with parallelization.
When i read this post: https://blog.runpod.io/runpod-serverless-pricing-update/, I have the impression that the cost is only linked to the “type of GPU” (16GB, 24GB, 48GB, ...) and that increasing the number of GPUs per worker doesn't increase the price per second. But that doesn't seem logical to me. Do I pay as much if I use a worker for 30s with 2 GPUs VS a worker for 30s with only 1 GPU? Or does the worker with 2 GPUs cost twice as much as the worker with a single GPU?
Also, when I read the doc: https://docs.runpod.io/serverless/references/endpoint-configurations#gpus--worker, it says that multi-GPU is only available on 48GB instances, but in the interface I get the impression that it's available on other types (the ones I'm interested in are 24GB). Is it just that the documentation isn't up to date, or is it a display problem?...
