Load balancing to death?
I've been sitting and watching your serverless system. And it just doesn't make sense.
I have two workers assigned. Yet you decide I need "Extra" instances spun up.
My workers are sitting there idle... oh there's a request let's send that to the "extra" queue. not the pod sitting idle....
Oh that last picture that was with an "Extra" pod we can't use that again we need to scrap that and use another cold booting pod...
Oh your workers are still idle shit .. well we better put them to sleep!
Oh you wanna downsize to one worker??? Cool well we are going to throttle that and stop it taking requests .. cool?
7 Replies
Unknown User•4mo ago
Message Not Public
Sign In & Join Server To View
Unknown User•4mo ago
Message Not Public
Sign In & Join Server To View
Trying to represent client's problem in a simpler way lol🤣. Let me know if it's representing something else than what's stated by op?
Unknown User•4mo ago
Message Not Public
Sign In & Join Server To View
Yeah, perhaps it wouldn't be so busy if a worker could keep its GPU for more than 30 seconds. I wonder how many times a GPu is shifted froma warm pod and given to a cold one? Guess I should count myself lucky .. I did have a good 20 min chat session last night without waiting 2 mins for a pod change... 😂
@Jason @CodingNinja follow up question though ... what is Throttled? .. A pod either has GPUs and is working, or it should be dead? My endpoint sitting "initializing" and not taking jobs for 20 mins is a slight problem right?
I've also noticed many "Throttled" "Extra" workers... so they are connected to my end point, when I don't need them, and they're not taking jobs anyway?
Unknown User•4mo ago
Message Not Public
Sign In & Join Server To View