Cuda error: illegal memory access encountered
I'm running stable diffusion webui in api mode on an A4500 (not serverless). Recently, I'm running into this issue a lot more frequently:
This wasn't happening before. Any idea what may be the issue here?
2023-12-29T00:37:16.100138705Z *** API error: POST: http://[redacted]/sdapi/v1/txt2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
2023-12-29T00:37:16.122794716Z Traceback (most recent call last):
2023-12-29T00:37:16.122830027Z File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
2023-12-29T00:37:16.122834167Z return self.receive_nowait()
2023-12-29T00:37:16.122837317Z File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
2023-12-29T00:37:16.122840327Z raise WouldBlock
2023-12-29T00:37:16.122843587Z anyio.WouldBlock2023-12-29T00:37:16.100138705Z *** API error: POST: http://[redacted]/sdapi/v1/txt2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
2023-12-29T00:37:16.122794716Z Traceback (most recent call last):
2023-12-29T00:37:16.122830027Z File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
2023-12-29T00:37:16.122834167Z return self.receive_nowait()
2023-12-29T00:37:16.122837317Z File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
2023-12-29T00:37:16.122840327Z raise WouldBlock
2023-12-29T00:37:16.122843587Z anyio.WouldBlockThis wasn't happening before. Any idea what may be the issue here?
