Software Engineering Courses (SECourses)•15mo ago

<@205854764540362752> Unfortunately after clean install - orch.OutOfMemoryError: CUDA out of memory.

@Furkan Gözükara SECourses Unfortunately after clean install - orch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.90 GiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 8.51 GiB is allocated by PyTorch, and 1.70 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. What else can I try? I did Clean VS2022 17.8.14, Nvidia Driver 561.09 game ready, cuda 12.4, WindowsInstallStep1 installer option #1, option #2 cudun, Windows_Update_kohya_fix_flux_step2, Windows_install_torch 2.5. Loaded 8GB config from dreambooth

TtheJindesan <@205854764540362752> Unfortunately after clean install - orch.OutOfMemoryError:...

Zet•10/25/24, 11:14 PM

How does your idle memory consumption look like?

Zet•10/25/24, 11:14 PM

ZZet How does your idle memory consumption look like?

theJindesanOP•10/25/24, 11:14 PM

Zet•10/25/24, 11:15 PM

let me fire up the config on my end

TtheJindesan <@205854764540362752> Unfortunately after clean install - orch.OutOfMemoryError:...

Furkan Gözükara SECourses•10/25/24, 11:17 PM

watch the task manager

Furkan Gözükara SECourses•10/25/24, 11:17 PM

and see if it uses shared vram

Furkan Gözükara SECourses•10/25/24, 11:17 PM

during trying to training

Furkan Gözükara SECourses•10/25/24, 11:18 PM

it has to use

FFurkan Gözükara SECourses watch the task manager

theJindesanOP•10/25/24, 11:18 PM

it does, almost full until it crashes.. Will screen cap. on next attempt

TtheJindesan it does, almost full until it crashes.. Will screen cap. on next attempt

Furkan Gözükara SECourses•10/25/24, 11:20 PM

it uses total 32.2 GB VRAM

Furkan Gözükara SECourses•10/25/24, 11:20 PM

it shouldnt crash it shows you have 40 GB

Furkan Gözükara SECourses•10/25/24, 11:20 PM

get a fresh config from the zip file

Furkan Gözükara SECourses•10/25/24, 11:20 PM

set only your training folder and model file paths

Zet•10/25/24, 11:24 PM

Also helps to close all applications and disable hardware acceleration for browser

Zet•10/25/24, 11:25 PM

let me reboot and do a stress test on the profile

FFurkan Gözükara SECourses it is 30 gb

Leolis78•10/25/24, 11:25 PM

Rank_3_T5_XXL_23500MB_11_35_Second_IT.json with V9 version on rtx3090, 8.94s/it

2024-10-25 20:14:29 INFO prepare CLIP-L for fp8: set to torch.float8_e4m3fn, set flux_train_network.py:509
embeddings to torch.bfloat16
INFO prepare T5XXL for fp8: set to torch.float8_e4m3fn, set embeddings flux_train_network.py:538
to torch.bfloat16, add hooks
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 50
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]2024-10-25 20:14:47 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1084
INFO text_encoder [0] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
INFO text_encoder [1] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090

epoch 1/200
INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▏ | 34/10000 [05:04<24:45:40, 8.94s/it, avr_loss=0.417]T

ZZet Also helps to close all applications and disable hardware acceleration for brows...

Leolis78•10/25/24, 11:30 PM

You can also temporarily enable the CUDA policy to use shared memory. And then calculate how much VRAM is missing because the CUDA error lies about the VRAM needed.

theJindesanOP•10/25/24, 11:31 PM