Cuda Driver Version issues
The error below shows up on various machines. I am using the following base images:
nvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04
nvidia/cuda:12.6.0-cudnn-devel-ubuntu22.04
nvidia/cuda:12.5.1-cudnn-devel-ubuntu22.04
Machine ID: lmrw66dp9b9b6r
For some reason when I change the cuda version, the error message changes based on the base version
cuda:12.8 shows cuda>=12.8
cuda:12.6 shows cuda>=12.6
cuda:12.5 shows cuda>=12.5
error starting container: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.8, please update your driver to a newer version, or use an earlier cuda container: unknown
11 Replies
I see this randomly from time to time. This machine also has the issue: 8akpfues8csotz
Oh, looks like the only cuda versions allowed on those machines are 12.4, that must be the issue. Any reason that Cuda Versions aren't kept somewhat up to date? 12.4 is a year old
if you use nvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04 then you should filter the pod that has cuda 12.8 in the host
the same thing like other version of base cuda image that you're using
select 12.8 here
Yeah, I found that, but is there a reason cuda versions aren't kept up to date?
i think it is to support older images, compability, stability and they are rolling update so not all machines are updated at the same time
Most cuda versions are pretty backwards compatible (if cuda 12.8 is available, as far back as 11.8 or farther should work fine in container), but I guess it probably comes down to not having the resourcing to update them super often
yeah true probably
if you want you can post a #🧐｜feedback about updating cuda in hosts and see what the staffs say
Thanks for your help!
your welcome
It used to be okay when the 4090 had been out for a while, but since 5090 only works with 12.8, it makes it very confusing for users on which template they should be using, they can't just use any one and have it work
i see, hmm what can we do about it