Runpod

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

Join

Runpod

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

Join

⚡｜serverless

⛅｜pods

🔧｜api-opensource

📡｜instant-clusters

🗂｜hub

swousy

8/1/2025

How do I set quantization to fp8 in the serverless settings?

If I select 'bitsandbytes' will that automatically change quant level to fp8?

AhmedElBana

8/1/2025

Store models in VRAM

Is there a possible way to store the models in VRAM as it keeps loading and ofloading each time I run a comfyui flow. Any suggestions?

SourabhSingh

8/1/2025

Setting up runpod serverless from scratch

Hello everyone, i am trying to deploy a comfyui severless endpoint to execute workflow that uses flux1-dev model to generate some model. I'm new to it and i have tried to create one serverless endpoint using Comfyui template from hub-listing. I already have a downloaded all the model in a network file along with all the custom nodes that i'll be using in the comfyui workflow that i want to execute. I did saw someone suggesting to make some changes in the handler.py script but i have no idea how to set that up. Can anyone guide me how can i set the serverless endpoint, because i have took the help of the documentation but those didn't help me and instead those document confused me....

Genia

8/1/2025

why the worker that is pulling docker image is considered running? and I probably pay for that time

bizetools

8/1/2025

RunPod Serverless Endpoint Issue - Jobs Complete But No Output Returned

Problem: My serverless endpoint jobs are completing successfully but returning empty results. Endpoint ID: su6ufhaephnw03 (Stable Diffusion XL) Symptoms:...

takezo07

7/31/2025

Serverless Endpoint with vLLM (Qwen2.5-VL-3B-Instruct)

I’m trying to set up a Serverless Endpoint on RunPod with vLLM (with Qwen2.5-VL-3B-Instruct). My goal is to get a lot of images descriptions. Here is how i set it up: Docker Image:...

ciiqmanu

7/31/2025

Build in pending for hours

Has anyone else experienced insanely long pending time before the build starts? This just happened to me a few days ago as is still happening. Every build is in pending state for hours or starts at very unpredictable times....

PotapovS

7/30/2025

5090 disappeared in Serverless

Hey 🖐 Is it just me, or has the 5090 disappeared when creating an endpoint? Or is it a temporary limitation due to lack of available resources?...

Kush

7/30/2025

Setting up a serverless endpoint for a custom model

Hi, complete beginner here. I've been trying to set up a custom model endpoint, but keep getting issues with having my requests answered (using the web application). I've gotten delay times of 40+ minutes so clearly something is wrong but I am having a hard time troubleshooting. Here are some details: - trained a model (base model deepseek-ai/DeepSeek-R1-Distill-Llama-8B) and saved to a huggingface repo - trying to publish endpoint via GitHub integration (already have folder containing Dockerfile, handler.py, requirements.txt)...

jcreus

7/29/2025

Regions with better/guaranteed bandwidth

Hi, I was interested in running serverless tasks but I need to upload the results of them reasonably fast (ideally in the 10 Gbps per gpu range). Are there regions that have faster or guaranteed bandwidth in the secure cloud, or is it possible to filter by that? Thanks!

red_wolf

7/29/2025

Updating CMD override after creating endpoint

I noticed that it's not possible to edit the CMD override after creating a serverless endpoint. I have to create an entirely new endpoint if I misconfigure / need to update this. Why?

red_wolf

7/29/2025

Configuring endpoints via API if publishing via Github integration

It seems that there's no way to create / update endpoints via the API, unless you host the images on a docker registry. Are there plans to extend API support to endpoints that are published from a github branch?...

saawhitelife

7/29/2025

s3 path for serverless image gen uploads (comfyui)

is it possible to set it via request body? if yes, how? we are using flux-schnel workflow

Solution:

https://docs.runpod.io/serverless/endpoints/send-requests#s3-compatible-storage-integration

Message Not Public

message.txt

AhmedElBana

7/29/2025

Most Available GPUs

Hi, I am using serverless with Network Volume attached to it, I need to know which region has the most available GPUs in the day, No matter what type of GPUs, The problem is my volume at US-CA-2 but most of the GPUs are not available and it usually initializing then throttled and most of the time unavailable, I know it's generic question and GPUs can't be trackable as they are shared but if anyone tried this before can help. Thank you....

Leo

7/29/2025

Serverless worker won't even start but counts as running

Hi so lately I've been dealing with various issues regarding serverless workers (i am using a custom docker file). At first i used the official base image with cuda 11.8 (and an older version of pytorch) and it worked fine with the 3090 but not with the 5090 (i have two serverless endpoints, one with "lower end gpus" and the other one with "higher end gpus"). So i used the base image with the latest version of everything (pytorch 2.7.1, cuda 12.9, ubuntu 24.04) but for some reason now the 3090 pod didn't work and the 5090 pod gave error CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected. I tried to tweak the docker image a bit but no success. Then i made the docker image install the package nvidia-cuda-toolkit and now, on the 5090 pod i get error CUDA error: no kernel image is available for execution on the device....

AhmedElBana

7/28/2025

Hi All,

Has anyone faced a problem reading some models like Pulid from Network Volume attached to the serverless? I am getting this error PulidFluxModelLoader 45: - Value not in list: pulid_file: 'pulid_flux_v0.9.1.safetensors' not in [] Any help please?...

billchen

7/28/2025

For 24G Pro Machines: Memory Allocation Must Not Be Less Than 60GB

I use the GPU of the 24 Pro, and the system allocates memory randomly. When the allocated memory is 46GB, I encounter an out-of-memory error. Could the allocated memory be at least 60GB or more? Please do not assign me a machine with 46GB of memory.

Ellroy

7/27/2025

Requests vs Jobs

On serverless, what's the relationship between a "request" and a "job"? I cancelled all of a worker's requests over 10 minutes ago and yet the logs show it still going (attached).

zongheng1619

7/25/2025

No cuda driver

RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.

Dan

7/25/2025

How to create serverless template

Hey guys. Where is the option to pick the type to be "serverless"??? I created a template and it is still not seen when i want to create a serverless endpoint

Previous Next

Gaming

Programming

Runpod

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

Runpod

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!