Someone is using my CUDA Memory?
Hi people, I get error when I try to train my model:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.06 GiB. GPU 0 has a total capacty of 19.67 GiB of which 30.31 MiB is free. Process 2169311 has 19.64 GiB memory in use. Of the allocated memory 19.43 GiB is allocated by PyTorch, and 22.72 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
I do not really see how, since I just booted the server, and the task is relatively simple, how is my memory fully used?
5 Replies
This happens with every pod I have btw, I have tried closing, re-opening new pods, getting bigger GPUS, my VRAM's are all gone
Unknown User•10mo ago
Message Not Public
Sign In & Join Server To View
@ege0189
Escalated To Zendesk
The thread has been escalated to Zendesk!
Yes, I have the same problem. Every pod which i wanna to start has less than 30% of memory.
Unknown User•10mo ago
Message Not Public
Sign In & Join Server To View