Runpod•2y ago•

61 replies

Maximum queue size

Hi, is there a limit for maximum pending jobs in the queue with serverless endpoints or are there any other queue size limitations?

Jason•6/27/24, 9:06 AM

i don't think theres any

digigoblin•6/27/24, 9:26 AM

There is a limit actually

digigoblin•6/27/24, 9:26 AM

100 * max workers

digigoblin•6/27/24, 9:28 AM

https://discord.com/channels/912829806415085598/948767517332107274/1203014459204050944

Jason•6/27/24, 9:36 AM

Oohh

jvm-cbOP•6/27/24, 9:48 AM

thx!

Ddigigoblin 100 * max workers

Encyrption•6/27/24, 12:12 PM

Ok, max workers = 100. Can I have 1000 API connection in the IN_QUEUE status? It's an exaggerated example but would like to know how many actual IN_QUEUE endpoint calls can be waiting? Say I have a serverless endpoint that receives 1000 calls but active workers is set to 0 and max workers is set to 1 and the process takes ~ 5 minutes. Can it process all 1000 calls, one at a time, while the remaining calls remain in the QUEUE?

EEncyrption Ok, max workers = 100. Can I have 1000 API connection in the IN_QUEUE status? ...

digigoblin•6/27/24, 12:13 PM

100 x 100 = 10,000 not 1000.

digigoblin•6/27/24, 12:14 PM

Read the link to flash-singh's message above, TTL also comes into play.

Encyrption•6/27/24, 12:17 PM

Ah so it all comes down to 'Idle Timeout' setting? What is the largest value the 'Idle Timeout' can use? 86,400 seconds would be 24 hours.

digigoblin•6/27/24, 12:17 PM

No it doesn't

digigoblin•6/27/24, 12:17 PM

if You have only a single max worker, you can only have 100 requests in the queue

digigoblin•6/27/24, 12:18 PM

simple mathematics

Encyrption•6/27/24, 12:18 PM

OK so with 1 active worker I can have a maximum of 99 IN_QUEUE connections waiting?

digigoblin•6/27/24, 12:19 PM

You should never have only 1 max worker under any circumstances anyway

Encyrption•6/27/24, 12:19 PM

with 1 max worker I don't spend any $ until my endpoint is being used.

digigoblin•6/27/24, 12:19 PM

I am not sure how the formula works with active workers, I assume the formula actually counts all workers not just max workers

digigoblin•6/27/24, 12:20 PM

@flash-singh would need to confirm though because I am just guessing, and @PatrickR should probably also document this info somewhere.

Encyrption•6/27/24, 12:21 PM

Ok thanks for the info you provided. Hopefully one of the people you have tagged will chime in.

digigoblin•6/27/24, 12:21 PM

Probably need them both to chime in because this question about how many max requests can be in queue has come up more than once so will be great if it can be documented.

Encyrption•6/27/24, 12:31 PM

I would hope that items can remain in the QUEUE for some time. All they are tiny amounts of information (JSON). Not sure why they would want to time them out.

Jason•6/27/24, 12:34 PM

Wdym

digigoblin•6/27/24, 12:35 PM

Its already been made clear that they don't, and it depends on your workers

digigoblin•6/27/24, 12:36 PM

Why would you want thousands of requests in the queue, that makes no sense

digigoblin•6/27/24, 12:37 PM

Usually when you have a large number of requests in the queue, you need to add more max workers to process the requests, or else you have some issue with your handler

Jason•6/27/24, 12:37 PM

Yeap, and if your request takes a long time to process you can use pods btw

digigoblin•6/27/24, 12:38 PM

Even if it takes long, you can still use serverless, just make sure you don't try and use 1 max worker

digigoblin•6/27/24, 12:38 PM

I want to rip my eyes out when people have 1 max worker and complain that things are not working as expected

Jason•6/27/24, 12:39 PM

Chill

digigoblin•6/27/24, 12:39 PM

I honestly don't get it, its not like you pay for them like active workers, so why set it to 1

digigoblin•6/27/24, 12:40 PM

And if you set it higher than 1, RunPod also gives you FREE "additional workers" to help with throttling

digigoblin•6/27/24, 12:41 PM

So there is absolutely no reason whatsoever to ever set it to 1. I don't even set it to 1 for debugging.

Ddigigoblin Its already been made clear that they don't, and it depends on your workers

Encyrption•6/27/24, 12:41 PM

I agree completely and I would do exactly that once I start getting users but before that I would like to save $ and not have active workers running when I have 0 users. But to understand what is possible in that initial phase I would need to know how many I can have in QUEUE. I would NEVER let there be 1000 in QUEUE but I could imagine a time where there is on average 10 or so in QUEUE.

EEncyrption I agree completely and I would do exactly that once I start getting users but be...

digigoblin•6/27/24, 12:42 PM

Just scale up max workers and you don't have a problem. I have my endpoint in production and have 30 max workers, zero active workers and never have any issues unless RunPod has issues.

digigoblin•6/27/24, 12:42 PM

I use network storage and sometimes there are networking issues and weird other incidents in those data centers.

Encyrption•6/27/24, 12:44 PM

With 0 active workers and 1 max worker what happens is nothing runs until an endpoint is hit. Once that happens a new serverless is spun up and responds. After that it goes back idle. If I had 1 active worker I would be charged for that endpoint sitting there waiting for requests right?

digigoblin•6/27/24, 12:44 PM

Please dont ever set max workers to 1.

digigoblin•6/27/24, 12:44 PM

I have been trying to make this clear in all my messages above, just DON'T do it.

Encyrption•6/27/24, 12:45 PM

Other than wanting me to pay more what is the reason?

Jason•6/27/24, 12:45 PM

Yeah bro just use more max workers, if you want you can set the scale type to the other one ( not queue delay )

digigoblin•6/27/24, 12:45 PM

Pay what more?

Jason•6/27/24, 12:45 PM

It doesn't make you pay more

digigoblin•6/27/24, 12:45 PM

Max workers are free

digigoblin•6/27/24, 12:45 PM

RunPod sets the default to 3 for a reason

Jason•6/27/24, 12:45 PM

Serverless charges from your running time only

digigoblin•6/27/24, 12:45 PM

They actually shouldn't allow you to set it less than 3 in my opinion.

Encyrption•6/27/24, 12:46 PM

Max workers, yes! I see. Could have 0 active workers and max of 3. That makes more sense for prod. That way I can handle the connections and again only being charged when used.

Jason•6/27/24, 12:46 PM

Yeah..

digigoblin•6/27/24, 12:46 PM

Exactly, like I said, I have 0 active workers and 30 max workers in production.

digigoblin•6/27/24, 12:46 PM

I don't pay a cent for active workers, I only pay when my max workers kick in and handle requests.

Maximum queue size

Similar Threads

Similar Threads

Similar Threads