OSError in vLLM worker; issues when its new update was released
I was using vLLM worker 1.7.0 and everything was working fine till yesterday. Today I am facing issues in all of my endpoints where huggingface models are deployed using the vLLM worker. Runpod logs shows OSError and the model cant be identified.
I then deployed a new endpoint with latest configuration of vLLM worker 1.9 and everything worked in the way it used to. @Justin Merrell
Runpod should let us know its changes atleast, so it does not affect the endpoints in production....
How much does it cost to use multi-GPU ?
I'd like to increase the number of GPUs per worker to get better performance with parallelization.
When i read this post: https://blog.runpod.io/runpod-serverless-pricing-update/, I have the impression that the cost is only linked to the “type of GPU” (16GB, 24GB, 48GB, ...) and that increasing the number of GPUs per worker doesn't increase the price per second. But that doesn't seem logical to me. Do I pay as much if I use a worker for 30s with 2 GPUs VS a worker for 30s with only 1 GPU? Or does the worker with 2 GPUs cost twice as much as the worker with a single GPU?
Also, when I read the doc: https://docs.runpod.io/serverless/references/endpoint-configurations#gpus--worker, it says that multi-GPU is only available on 48GB instances, but in the interface I get the impression that it's available on other types (the ones I'm interested in are 24GB). Is it just that the documentation isn't up to date, or is it a display problem?...
Process group has not been destroyed before destruct ProcessGroupNCCL, Leaked shared_memory object
Is this an issue that i need to be concerned about ?
Serveless UI broken for some endpoints
Since the latest UI changes, clicking on some endpoints will create a constant loading of the runpod logo and the UI never loads. This seems to be only happening with certain enpoints.
Need help in fixing long running deployments in serverless vLLM
Hi, I am trying to deploy migtissera/Tess-3-Mistral-Large-2-123B model on serverless using 8 48GB GPUs using vLLM.
The total size of model weights around 245 GB.
I have tried two ways: 1st way without any network volume, it takes really long time to serve first request as it needs to download the weights. Then if the worker goes to idle and I sent a request again it downloads weights and takes long time....
delayTime representing negative value
On some requests I started to see delayTime in negative and this affects my own autoscaler.
In Faster whisper serverless endpoint, how do i get english transcription for tamil audio
In Faster whisper serverless endpoint, how do i get english transcription for tamil audio. When i test it with tamil audio, I get output like this, how do I get it in English.
All GPU unavailable
I just started using RunPod. Yesterday, I created my first serverless endpoint and submitted a job, but I didn't receive a response. When I investigated the issue, I found that all GPUs were unavailable. The situation hasn't changed since then. Could you tell me what I should do?
