Serverless Worker Fails: “cuda>=12.6” Requirement Error on RTX 4090 (with Network Storage)

Hi team, Hi team, I’m deploying a serverless worker that requires CUDA 12.6 or higher. My endpoint is set to use RTX 4090 GPUs, and I’m using a network volume for model storage (models are not baked into the image). However, when the worker starts, I get this error: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.6, please update your driver to a newer version, or use an earlier cuda container: unknown My endpoint is configured to require CUDA 12.6+. I’m using network storage for models, as recommended for large model workflows. The error persists even though the GPU should be compatible. Questions: Is this due to the host’s NVIDIA driver version being too old, even though the GPU is compatible? How can I ensure my jobs are only scheduled on hosts with drivers that support CUDA 12.6 or higher? Is there a way to avoid this error or improve scheduling reliability for CUDA 12.6+ workloads with network storage? Thanks for your help!