Harware Side issue: Cuda Error ECC
Traceback (most recent call last):
File "/workspace/ComfyUI/main.py", line 179, in <module>
import execution
File "/workspace/ComfyUI/execution.py", line 18, in <module>
import comfy.model_management
File "/workspace/ComfyUI/comfy/model_management.py", line 251, in <module>
total_vram = get_total_memory(get_torch_device()) / (1024 * 1024)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/model_management.py", line 236, in get_totalmemory
, mem_total_cuda = torch.cuda.mem_get_info(dev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/venv/lib/python3.12/site-packages/torch/cuda/memory.py", line 897, in mem_get_info
return torch.cuda.cudart().cudaMemGetInfo(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: uncorrectable ECC error encountered
Search for
I am getting this traceback on RTX 6000 PRO Pods on US-NC-2
i have never seen anything like this. Seems to be a hardware side issue rather than a software one.
File "/workspace/ComfyUI/main.py", line 179, in <module>
import execution
File "/workspace/ComfyUI/execution.py", line 18, in <module>
import comfy.model_management
File "/workspace/ComfyUI/comfy/model_management.py", line 251, in <module>
total_vram = get_total_memory(get_torch_device()) / (1024 * 1024)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/model_management.py", line 236, in get_totalmemory
, mem_total_cuda = torch.cuda.mem_get_info(dev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/venv/lib/python3.12/site-packages/torch/cuda/memory.py", line 897, in mem_get_info
return torch.cuda.cudart().cudaMemGetInfo(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: uncorrectable ECC error encountered
Search for
cudaErrorECCUncorrectable' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with I am getting this traceback on RTX 6000 PRO Pods on US-NC-2
i have never seen anything like this. Seems to be a hardware side issue rather than a software one.
