R
RunPod•4mo ago
Yannik

best practice to terminate pods on job completion

I have a one time job I want to run as a GPU pod. Currently the container gets restarted as soon as the job finishes. What's the best way to terminate the pod after completion?
6 Replies
ashleyk
ashleyk•4mo ago
You can use runpodctl to terminate the pod. 
runpodctl remove pod
runpodctl remove pod
https://github.com/runpod/runpodctl
GitHub
GitHub - runpod/runpodctl: 🧰 | RunPod CLI for pod management
🧰 | RunPod CLI for pod management. Contribute to runpod/runpodctl development by creating an account on GitHub.
Yannik
Yannik•4mo ago
Thanks for the pointer. You mean I should install it in the container and add my credentials to it, and then terminate as soon as the job is done?
ashleyk
ashleyk•4mo ago
You may also want to consider using serverless instead of GPU cloud.
Yannik
Yannik•4mo ago
I don't have a large number of jobs (~20) each running for ~1h. Does that still make sense?
justin
justin•4mo ago
Yeah, I think this is fine. Whatever you have running on serverless, is essentially what could be running on GPU Cloud, except that you are just calling a handler.py instead that calls the runpod.start() method. can check their documentation. I should really improve this lol, but it should give you a decent pseudo example: https://discord.com/channels/912829806415085598/1194695853026328626 I also run jobs every so often like transcriptions on runpod, but I dont want to be spinning up a pod every time So I just ping it to a serverless function to queue up and wait for the responses to finish GPU Cloud the difference is usually u start up some openssh server and jupyter lab, but the environment is the same setup if it works in GPU Cloud, will work in serverless (most of the time with some exceptions)
Yannik
Yannik•4mo ago
Okay awesome, thank you for the link to the how to thread. I will give it a look
Want results from more Discord servers?
Add your server
More Posts
Run LLM Model on Runpod ServerlessHi There, I have LLM Model which build on docker image and it was 40GB++ docker Image. I'm wonderiSafetensor safeopen OS Error device not foundRunning inference on severless endpoint and this line of code: ``` with safetensors.safe_open(path, Can I turn off few vCPU?I'm doing an overload test, but I don't know how to limit vCPU in Python.Deploying H2O LLM Studio /w auth using NgrokI have been working most of the day to get this container deployed to runpod. Here's the trick thougL40 and 6000 Ada serverless worker not spawningIt is not spawningWrong GPUs being assignedI'm paying for the "7 x H100 80GB SXM5", but when I run nvidia-smi I get that there's actually 7 NVIDirecting requests from the same user to the same workerGuys, thank you for your work. We are enjoying your platform. I have the following workflow. On thServerless webhook for executionTimeoutHi, We've just added an executionTimeout for our serverless jobs. I understand that when you supply Is there any way to do dynamic batching?Say I have a vision model deployed and I send 5 images within x time is there a way to actually stacNetwork Volume suddenly empty in EU-RO-1After a restart of a CPU pod that was attached to a network volume located in EU-RO-1 region, the /w