Runpod•10mo ago

queue delay times

Hi , I'm seeing really long delay times . even though there's nothing in the queue , and this is a really small CPU serverless endpoint . Any idea what causes this ?

youssef•2/25/25, 9:34 PM

I'm having same issue, on 16gb/24gb, my request stays a shit ton of time in the queue, these are only two items

Capture_decran_2025-02-26_a_01.33.50.png

youssef•2/25/25, 9:34 PM

cc @PRB - any issue on the statuspage ?

PRB•2/26/25, 4:34 PM

@sahir which datacenter are you running this in?

PRB•2/26/25, 4:35 PM

@youssef can you open a support ticket? we are already looking at this but will be nice to keep track and get back to you

PPRB @sahir which datacenter are you running this in?

sahirOP•2/27/25, 3:52 AM

all locations were selected , so its made workers here

sahirOP•2/27/25, 6:19 AM

This is happening to my other endpoints too now

Kays•2/27/25, 3:40 PM

Same here, almost 2 minutes cold start every time

Kays•2/27/25, 3:41 PM

but once every few requests it goes <5 seconds again

PRB•2/27/25, 5:37 PM

are you guys on cpu endpoints or GPU endpoints?

PRB•2/27/25, 5:38 PM

@Kays please reply so i can resolve faster

Kays•2/27/25, 5:39 PM

GPU endpoints

Kays•2/27/25, 5:39 PM

Its mostly A100 for me

Kays•2/27/25, 5:39 PM

I mean h100*

PRB•2/27/25, 5:43 PM

endpoint id will help

Kays•2/27/25, 5:52 PM

pury32p7r6r4wf

Kays•2/27/25, 5:53 PM

I can give you an example test request if you like

KKays pury32p7r6r4wf

flash-singh•2/27/25, 6:40 PM

your cold starts are high, are you loading model from network volume or is the model just too big?

Fflash-singh your cold starts are high, are you loading model from network volume or is the m...

Kays•2/27/25, 6:42 PM

I'm not using network volumes, the model is flux-dev (24gb)

Kays•2/27/25, 6:42 PM

But what's weird is that cold start sometimes is extremely quick, like under 5 seconds

Kays•2/28/25, 2:02 PM

Hey there, any updates on this? Is it just the model being too big? @PRB @flash-singh thanks!

Kays•2/28/25, 5:33 PM

seems to be fixed now somehow

KKays But what's weird is that cold start sometimes is extremely quick, like under 5 s...

flash-singh•2/28/25, 5:54 PM

thats just flashboot, anything over 10s should be your ideal cold start, is your model baked into the container image?

Kays•2/28/25, 7:22 PM

yes it is on the container

Kays•2/28/25, 7:22 PM

right now I'm getting around 50/50 flashboots

flash-singh•2/28/25, 11:50 PM

thats about right, depends on workload and capacity, for h100s thats really good if your p50 is hitting flashboot

Kays•3/1/25, 12:18 AM

Cool, yes I’m happy with that rate, yesterday was more like 90-10 that’s why I mentioned it

queue delay times

Similar Threads

Similar Threads

Similar Threads