Runpod•4mo ago

Workers are getting throttled

Hey guys,

Workers are getting throttled. I have 50 workers limit and most of them are getting throttled. My application is being impacted havily.

For a note, its mostly happening for US based workers. I have no preferences around GPU or CUDA so its starting worker randomly across the globe.

J.•9/5/25, 8:47 PM

Hey @Jaya do you have an endpoint id you can share?

JayaOP•9/5/25, 9:13 PM

I have many endpoints. One of them is

https://console.runpod.io/serverless/user/endpoint/2nbf2hwyiledzn

J.•9/5/25, 9:21 PM

@Jaya what i see is that your GPU is only set to 4090, or the 24gb pro, if you are okay with doing so / your application can handle it, i can recommend to also maybe allow for a 24gb in your selection for endpoint.

4090s are quite popular and can be eaten up. You can also request for a larger worker max if you feel that is helpful to increase your workers across your endpoints

JayaOP•9/5/25, 9:23 PM

Ok can you help me with a large worker max? I will also have 24gb has secondary but honestly speaking its just started since last couple of days

JayaOP•9/5/25, 9:23 PM

It wasnt that bad so far

J.•9/5/25, 9:28 PM

@Jaya i increased your max workers to 75, to give you a bit more wiggle room on any critical endpoints so feel free to bump that up, if you need even more workers, you can fill out a hubspot form, in the UI at your workers amount if you need more you can click it, and it will bring you to a form to update even more

JayaOP•9/5/25, 11:13 PM

Thanks Justin

Xeverian•9/6/25, 7:21 PM

Can confirm that it doesn't look good last several days (was fine before). I also use 4090s

PotapovS•9/8/25, 6:18 PM

Hey

I'm experiencing the same issue. Almost all workers on 4090 are showing as throttled with Low Supply.
The 5090s are completely unavailable.

XXeverian Can confirm that it doesn't look good last several days (was fine before). I als...

J.•9/8/25, 6:26 PM

Just to confirm, this is without region restrictions?

PPotapovS Hey 🖐 I'm experiencing the same issue. Almost all workers on 4090 are showing a...

J.•9/8/25, 6:27 PM

Is this also no region restrictions?

JJ.Is this also no region restrictions?

PotapovS•9/8/25, 6:30 PM

Yes. Region doesn't matter.

J.•9/8/25, 6:32 PM

Yeah, 4090s and 5090s might just be being used right now popular GPUs. but raised it to the team to see if any further concerns, but this can happen especially if everyone is concentrating on these two GPU types

PotapovS•9/8/25, 6:34 PM

Thanks for answer. Please help. Because GPUs don't work at all.
Constant throttled.

PPotapovS Thanks for answer. Please help. Because GPUs don't work at all. Constant throttl...

J.•9/8/25, 6:34 PM

Can you share with me the endpoint? is your entire endpoint throttled?

J.•9/8/25, 6:34 PM

It should still have left some to use

PotapovS•9/8/25, 6:36 PM

The entire endpoint is throttled.
I think the id is not important. You can create a point yourself now and check that 4090 is almost unavailable. 5090 is completely unavailable.

PotapovS•9/8/25, 6:37 PM

I see that workers are initialized, but then immediately become throttled.

J.•9/8/25, 6:39 PM

Will test and raise to the team, thanks for reporting

J.•9/8/25, 8:07 PM

Reported to the teeam @PotapovS / @Xeverian / @Jaya is being tracked + worked on fyi

J.•9/8/25, 8:07 PM

Thanks for helping to report the issue

JayaOP•9/8/25, 8:57 PM

This does not seem to be improving @justin (New) [Staff Not Staff] . I have also a 48 GB A40, A6000 but there is none which can take place

JJaya This does not seem to be improving @justin (New) [Staff Not Staff] . I have also...

J.•9/8/25, 9:10 PM

Maybe can set the other 48gb as higher priority for now in the menu, but yes, I’ve already raised this as a high priority and they identified the potential issue and are rolling some stuff back and changes are currently underway

J.•9/9/25, 3:13 PM

It should be better now today FYI: 4090s and 5090s issue were resolved yesterday night it seemed. Thanks again to everyone for pointing out the issue.

JJ.It should be better now today FYI: 4090s and 5090s issue were resolved yesterday...

PotapovS•9/10/25, 1:46 PM

I confirm. The problems have been resolved.
Thank you and the team for your help!

JayaOP•9/16/25, 11:04 AM

Now there is another issue. Requests are taking way too long time to move from in queue to in progress.

JayaOP•9/16/25, 11:05 AM

And this time I have it as H100. It seems the problematic ones are those running in North America and are unable to reserve GPUs

JayaOP•9/16/25, 11:52 AM

It seems its a cuda error. I have no preferences around cuda. This has been giving lot of trouble recently.

JJaya And this time I have it as H100. It seems the problematic ones are those running...

J.•9/16/25, 1:43 PM

Sorry to hear, feel free to create a support ticket:
https://contact.runpod.io/hc/en-us/requests/new

And will be more easily escalted to the right team / points of contacts who can look deeper into it

JJ.Sorry to hear, feel free to create a support ticket: https://contact.runpod.io/h...

PotapovS•9/26/25, 5:21 PM

Hey

Problem with throttled workers back.
All regions, all cuda versions. 5090 constantly falls off and throttled.

PPotapovS Hey 🖐 Problem with throttled workers back. All regions, all cuda versions. 5090...

JayaOP•9/26/25, 9:21 PM

I can confirm its back with 4090 as well

Hidobo•10/4/25, 8:13 AM

5090 completely unusable

JJ.Sorry to hear, feel free to create a support ticket: https://contact.runpod.io/h...

PotapovS•10/6/25, 9:46 PM

Hey

Problem with throttled workers back again.
All regions, all cuda versions. 5090 constantly falls off and throttled.

PPotapovS Hey 🖐 Problem with throttled workers back again. All regions, all cuda version...

J.•10/6/25, 9:54 PM

Thank you, will be taking a look and raising to the team

PPotapovS Hey 🖐 Problem with throttled workers back again. All regions, all cuda version...

J.•10/6/25, 10:04 PM

Thanks have confirmed - there is extremely high usage right now from someone eating up GPUs. Flagged to the team

Jackie•10/6/25, 10:26 PM

5090's are super scuffed rn

J.•10/6/25, 10:27 PM

Yes; to give an update we just have extremely high utilization right now from all customers maxing out the data-centers. Team is planning to increase capacity in upcoming 1-2 weeks since takes time to get physical hardware online

WeamonZ•10/6/25, 10:49 PM

Not a single one of my endpoints have a 5090 available

Alex Bruskin•10/8/25, 6:28 AM

I want to comment on the throttling issue. For us it started a few days ago, and it is always happening around 11am EST. It feels like someone starts running something big, pushing everyone else out. We tried data centers in Iceland and Romania, it is all the same.

JayaOP•10/15/25, 1:10 PM

This is back again in last few days. Most of the workers getting killed for 4090. Also, some times it happens with H100 as well

JayaOP•10/15/25, 1:11 PM

This is highly disappointing.

JayaOP•10/15/25, 1:11 PM

It seems you cant run a business with assumptions you will get serverless GPUs on runpod.

Dj•10/15/25, 2:18 PM

Throughout this week we've been running emergency maintenance and the users most affected are those running serverless workloads with popular/low cost GPUs. Where we may have a surplus of a specific GPU, we have to delist those machines to perform work on them. We are obligated to perform this maintenance across the fleet and only ask for your patience until it's done and we can disclose the reason.

JayaOP•10/15/25, 3:37 PM

Your reason is absolutely justified but you have to consider this fact that other businesses are relying on GPUs provided by you.

Xeverian•10/15/25, 8:36 PM

: (

Xeverian•10/15/25, 8:40 PM

Xeverian•10/15/25, 8:41 PM

and it's 4090 in EU-RO-1, the best one for those cards

Workers are getting throttled

Similar Threads

Similar Threads

Similar Threads