R
RunPod•4mo ago
Casper.

2x A100 / 3x 48 GB on Serverless

Hi @flash-singh, a while back we talked about having multiple GPUs on serverless and then you introduced 2x 48 GB. Now there are larger models out like Mixtral 8x7B which requires a minimum of 100GB, but ideally 120GB VRAM to serve. Do you have any plans to expand capacity to allow for this in your serverless products? Perhaps, an easier route is to allow 3x 48 GB GPUs since that can serve models like Mixtral.
3 Replies
flash-singh
flash-singh•4mo ago
MI300 are coming in April, until then we might allow 2x A100 and 3x A6000
Casper.
Casper.•4mo ago
MI300 will be nice. At what price though?
flash-singh
flash-singh•4mo ago
likely similar to H100 price
Want results from more Discord servers?
Add your server
More Posts
How do I run custom code on a Runpod instance?This sounds like you need to create a publicly accessible Docker image to run any code, is that trueRunning on local URL but can't access from outsideHi guys, I have this service running in my pod but I can't seem to access it through the intended waHow can I use ollama Docker image?Hello. I've been trying to serve ollama on RunPod using ollama Docker image (https://hub.docker.com/Comfyui won't run because of the missing NVidia driversyesterday my comfyui worked well, now when i try to launch it i get the error: RuntimeError: Found nRunPod Library + APISo I am attempting to create an API to either start/stop an existing pod or create a pod and then stCuda DriverI am getting the following error. I was setting up a new runpod, since the one I setup last night waSGLang worker (similar to worker-vllm)Recently, some progress has been made for efficiently deploying LLMs and LMMs. SGLang is up to 5x faVery low download speed. Will take days to download the model300 Kb/S IS NOT FAIR!!!