Runpod•9mo ago

Problem with hanging pod

Hi team, I've issue about the hanging pod, somehow the GPU is crashed and now all the process is hanging

Tried to restart the pod, it didn't work. Tried to stop and start again, and now it's can't get the pod up

Please help me with this. This is the pod ID:
whainytdwrgb7l

ferdinandjasonOP•4/10/25, 2:26 PM

https://contact.runpod.io/hc/en-us/requests/15833
Raised in there also

ferdinandjasonOP•4/10/25, 2:29 PM

Now it's getting this

Jason•4/10/25, 3:16 PM

oh maybe there's some problem with the pod then

ferdinandjasonOP•4/10/25, 3:19 PM

can up now, but it's restarted / stopped multiple times

ferdinandjasonOP•4/10/25, 3:19 PM

it's really disturbing right now

ferdinandjasonOP•4/10/25, 3:21 PM

and now Problem with hanging pod it's happening again

ferdinandjasonOP•4/10/25, 3:21 PM

and I'm still charged

ferdinandjasonOP•4/10/25, 3:23 PM

now stopped the pod

Jason•4/10/25, 3:24 PM

hmm

Jason•4/10/25, 3:24 PM

what did you run

ferdinandjasonOP•4/10/25, 3:28 PM

I'm training machine learning model right now, so some python scripts

Jason•4/10/25, 3:30 PM

Any error message related to the "gpu crash"?

Jason•4/10/25, 3:30 PM

Or what does that means?

Jason•4/10/25, 3:31 PM

Might be related to too much ram used ( full), you might need higher pod ram

ferdinandjasonOP•4/10/25, 3:31 PM

no error message related sadly

the training loop is somehow stuck, and not going any further,
and when running nvidia-sminvidia-smi command it's still hangging around 7 hours

ferdinandjasonOP•4/10/25, 3:32 PM

the VRAM is around 66% usages

Jason•4/10/25, 3:32 PM

I see

Jason•4/10/25, 3:32 PM

The ram?

JJason The ram?

Jason•4/10/25, 3:32 PM

What about the ram

Jason•4/10/25, 3:32 PM

I'd suggest moving and trying in another pod

Jason•4/10/25, 3:33 PM

But surely you can report this to support ticket, have you created one?

ferdinandjasonOP•4/10/25, 3:33 PM

from the statistic, the ram is only 2% at max

ferdinandjasonOP•4/10/25, 3:34 PM

I'd suggest moving and trying in another pod

I still need the data / the training checkpoint, how do to that?

ferdinandjasonOP•4/10/25, 3:34 PM

But surely you can report this to support ticket, have you created one?

created in this https://contact.runpod.io/hc/en-us/requests/15833

Jason•4/10/25, 3:35 PM

Okayy

Fferdinandjason > I'd suggest moving and trying in another pod I still need the data / the train...

Jason•4/10/25, 3:36 PM

Ohh, you dont have any backup now?

Jason•4/10/25, 3:36 PM

Can you access the terminal and run commands then?

Jason•4/10/25, 3:37 PM

If yes you can install any file transfer tool that can be used over the internet

Jason•4/10/25, 3:37 PM

Like rsync rclone, syncthing

Jason•4/10/25, 3:38 PM

Then move the data to the other pod

Jason•4/10/25, 3:38 PM

Try to restart the pod if you got your Data in your volume ( mounted path)

ferdinandjasonOP•4/10/25, 3:39 PM

yes, I don't have any backup right now

tried to start the pod again ~10 minutes ago, but it's not getting up until right now

Jason•4/10/25, 3:41 PM

What's the logs right rn

ferdinandjasonOP•4/10/25, 4:35 PM

no logs

Jason•4/11/25, 12:09 AM

Well what template are you using

Jason•4/11/25, 12:10 AM

Maybe the pod is down, just wait for support then

Fferdinandjason no logs

riverfog7•4/11/25, 4:59 AM

Maybe look at wandb logs

riverfog7•4/11/25, 4:59 AM

And there should be a copy of ur code in wandb

Problem with hanging pod

Similar Threads

Similar Threads

Similar Threads