How does runpod handle pod terminating
It is very likely that runpod simply sends a sigkill to the main container process. This is really annoying when you are trying to handle termination. Could you please provide information on how your orche system handles pod termination and how I can get the OS signal
16 Replies
Just to mention: kubernetes sends SIGTERM
It is crucial for us, because of LLM streaming. We wanna utilize some big GPUs, but it is not really possible, because any progressing stream will be terminated no matter what 😦
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
Yes, it is, but while stream is in the progress - it would be dropped without any graceful period. My point is that the pod should have a graceful period instead of an instant sigkill
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
on demand
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
yeah you right
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
for a graceful minute
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
pod receives termination -> sigterm -> 1min alive (graceful period) -> pod sends sigkill and dies
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
Because kubernetes does that way and it allows pod to handle graceful term, instead of instant annihillation (runpod does that way)
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View
I appreciate your advice, I'll send it as feedback tho
Unknown User•15mo ago
Message Not Public
Sign In & Join Server To View