R
RunPod•4mo ago
Antoine Dangeard

Help with constantly crashing GPU pods

Hello, I’ve been struggling for the past few days with trying to get a docker image up and running on a GPU pod. I had success with a template I made (docker image mcgillrobotics/mujoco:cuda118) and managed to connect and get things running, but since then I have not been able to successfully connect to a pod. The docker image pulls, but when I click “Connect to web terminal” nothing happens. When I try to SSH it says the container is not running and kicks me out instantly. I’ve tried different rocket images, different CUDA versions, GPUs, template overrides but have had no luck. I reached out to support on the website and was told I would receive an email, but it’s been a few days and I’ve heard nothing. Would be super grateful if anyone has any input!
Solution:
Yes you need to add sleep infinity
Jump to solution
4 Replies
ashleyk
ashleyk•4mo ago
Sounds like your Docker image isn't keeping the container alive
Antoine Dangeard
Antoine Dangeard•4mo ago
What do you mean “keeping it alive”? Is there something special I need to do so the container doesn’t die? Or a crash or something?
Solution
ashleyk
ashleyk•4mo ago
Yes you need to add sleep infinity
Antoine Dangeard
Antoine Dangeard•4mo ago
gotcha, thanks!
Want results from more Discord servers?
Add your server
More Posts
[Urgent] failed : Software caused connection abortCan someone help with this error please? it's causing us a huge problem with our next release. Tryhow to distribute usage of GPUI purchased 2 RTX A5000 runpods. but my server is using only 1 GPU on 99%. other gpu is 0. is this wServerless not returning errorThe following code: ``` def handler(event): try: logger.info('validating input') Converting to Team AccountHi, will my history usage and billing be public to my team if I convert my personal account into a tGetting 404 error when making request to serverless endpointI'm using the python SDK, and pasting in the endpoint ID into the provided example code. Here is theout of memory errorout of memory error GPUterminalCant connect to web terminal or ssh to podCompatibility of RTX A6000 for Multi-GPU TrainingI would like to inquire about the types of GPUs that support multi-GPU training. For instance, is itOut of memory errors on 48gb gpu which didn't happen beforeSome requests fail due to OOM, but the endpoint uses 48gb and is definitely capable of processing thH100 multi-gpus settingsWhen I tried to load weights from checkpoints on my custom model using multi-gpus, weights are not l