Cuda Driver Version issues
The error below shows up on various machines. I am using the following base images:
nvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04
nvidia/cuda:12.6.0-cudnn-devel-ubuntu22.04
nvidia/cuda:12.5.1-cudnn-devel-ubuntu22.04
Machine ID: lmrw66dp9b9b6r
For some reason when I change the cuda version, the error message changes based on the base version
cuda:12.8 shows cuda>=12.8
cuda:12.6 shows cuda>=12.6
cuda:12.5 shows cuda>=12.5
error starting container: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.8, please update your driver to a newer version, or use an earlier cuda container: unknown
10 Replies
I see this randomly from time to time. This machine also has the issue: 8akpfues8csotz
Oh, looks like the only cuda versions allowed on those machines are 12.4, that must be the issue. Any reason that Cuda Versions aren't kept somewhat up to date? 12.4 is a year old
Unknown User•6mo ago
Message Not Public
Sign In & Join Server To View
Yeah, I found that, but is there a reason cuda versions aren't kept up to date?
Unknown User•6mo ago
Message Not Public
Sign In & Join Server To View
Most cuda versions are pretty backwards compatible (if cuda 12.8 is available, as far back as 11.8 or farther should work fine in container), but I guess it probably comes down to not having the resourcing to update them super often
Unknown User•6mo ago
Message Not Public
Sign In & Join Server To View
Thanks for your help!
Unknown User•6mo ago
Message Not Public
Sign In & Join Server To View
It used to be okay when the 4090 had been out for a while, but since 5090 only works with 12.8, it makes it very confusing for users on which template they should be using, they can't just use any one and have it work
Unknown User•6mo ago
Message Not Public
Sign In & Join Server To View