Comfyui + RTX 5090 w/ Nvidia Driver 570 Issue
Most of the RTX 5090 GPUs aren't running with driver 572+, they're running with 570, and the 5090 really doesn't like that configuration for some reason (cu128+, pytorch2.7.1+, 570). Image generation workloads easily take twice as long as running on a 6000 Ada with the same docker image, and this is a somewhat known issue. While US-CA-2 has RTX 5090s with the necessary driver version (btw this is not something you can set with an image...it's entirely dictated by RunPod / the datacenter), most other regions do not and the 5090 shouldn't generally be recommended for use with ComfyUI.
7 Replies
Interesting, does it care specifically for that driver number? Can you share a little detail about why this is an issue? @trillagodmode
It's an issue because if you try and use what would otherwise be an appropriate image for your pod, cu128 and pytorch2.7.1 and python3.12 for example, you stiill won't be able to utilize the rtx5090 properly because it needs 572+ for the driver. without it some kernel nonsense goes on and the 5090 is actually slower than using a 4090, for example. so obviously if a user isn't aware of this and is trying to use a more premium GPU, they'll have an even worse experience than if they used a 4090 or 6000 Ada or whatever else
I'm a little confused in that all of the 570 family should be CUDA 12.8 compatible. If you're on Windows the numbers are a little higher bc windows driver versions don't match exactly though.
CUDA 12.8 released with Linux R570 and Windows R571 as a requirement
https://docs.nvidia.com/cuda/archive/12.8.0/cuda-toolkit-release-notes/index.html#cuda-toolkit-major-component-versions
(Search the table for "NVIDIA Linux Driver")
1. CUDA 12.8 Release Notes — Release Notes 12.8 documentation
The Release Notes for the CUDA Toolkit.
And CUDA 12.8 U1 released with Linux R570 and Windows R572 :fbslightsmile:
https://docs.nvidia.com/cuda/archive/12.8.1/cuda-toolkit-release-notes/index.html#cuda-toolkit-major-component-versions
1. CUDA 12.8 Update 1 Release Notes — Release Notes 12.8 document...
The Release Notes for the CUDA Toolkit.
Do you mind sharing the error you're seeing? Also just as an aside we're looking to upgrade all of the fleet very soon so things like this will just never be an issue.
exactly, 572 is the floor like i said
the 5090 on US-IL-1 have 570 when you do nvidia-smi
wait let me look at my notes
yeah the US-IL-1 rtx5090s are using Driver Version: 570.124.06
from what i found online 575 is where the drivers get really stable for 5090s...so certain comfyui nodes, like ultimate sd upscale, just do not work properly in that environment, taking easily 5x longer than a 4090 or 6000 Ada. the second i switched over to a region with 12.9, which necessitates 575+, the 5090 was absolutely crushing it