Stable Diffusion Stopped Working After a Restart
Stable Diffusion was running fine until it stopped (unfortunately I get no notification to put funds back on the account). But after restarting it there was an odd error. I made no changes that I know of. I am a noob and I didn't even set this up. How do I find help in getting this back up and running?
Connectivity issue on 4090 pod
Hello Runpod,
I've been unable to access a stopped 4090 pod for quite some time now (approx 10-12 hrs). The pod ID is 24kw7y5uu2yuil, in the
IS datacenter. During this time, the attached notice about a network outage has been displayed for that pod, and the process to launch the pod gets stuck at Waiting for logs as in the second attached image. This happens when trying to launch with any number of the pod's GPUs (0-8 inclusive). I do not need to use this pod's GPUs but I do have some important data I need to transfer from it. I've been waiting to post something about this since I've been assuming the network issue is transient, but as it's been happening since before I went to bed last night, I figured I would reach out to see if there's any way I can get the data off of this pod.
Thanks!...
P2P is disabled between NVLINK connected GPUs 1 and 0
Hey team! Could you fix NVLink issue for H100 SXM Community pods? I encounter this error frequently. Corrupted pod ID: 4a5acwxj2kene6
P2P is disabled between NVLINK connected GPUs 1 and 0. This should not be the case given their connectivity, and is probably due to a hardware issue. If you still want to proceed, you can set NCCL_IGNORE_DISABLED_P2P=1.
I can proceed with NCCL_IGNORE_DISABLED_P2P flag but this will drop performance ~ 10%...
Solution:
@storuky2306 so got response and aparently gpu5 is not supporting P2P.
What we can advise for now is to pick diffrent machine...
Pod Downsized, with Pictures
I have been having issues with the A6000's, my pods keep getting downsized. I wasn't able to catch a snip of the first couple times but I did here. As you can see I am renting 2 A6000's but when I try to start it I am only renting 1. What is happening here? this keeps on happening so I have to terminate and start new pods repeatedly, my entire audit logs is me restarting new A6000's and I keep losing money trying to get the service to give me what I bought. (ID:wp0ofb7xs94uf0)
Solution:
Hi @MushyPotato - when you rent a pod on spot and it gets stopped (really, when a pod gets stopped for any reason) the GPUs in that machine are made available for other customers, and by the time you can start it again there may be fewer GPUs available.
Setting up a network volume will allow you to deploy to any GPU within that data center, so you will not be limited by whatever the GPU rental status in your specific machine is....
Question about graphql API
In https://doc.runpod.io/recipes/view-gpu-types-info
If securePrice is zero does that mean that the resource is not available?...
Solution:
@ChD That's correct - likely the A100 SXMs were all in use at the time that you ran the query, globally out of all specific GPU specs they're the most likely to go completely unavailable at the moment
There are some now and it's showing the price for me: "id":"NVIDIA A100-SXM4-80GB","securePrice":2.29}...
