I was running on a 5090 pod with 3 GPUs (that's what was available to it). Mid-run my software complained that there are no CUDA GPUs. After stopping the app I tried nvidia-smi and got "Failed to initialize NVML: Unknown Error". That's never happened before.