How does your idle memory consumption look like?
How does your idle memory consumption look like?
Z
T
Zlet me fire up the config on my end
Fwatch the task manager
Fand see if it uses shared vram
Fduring trying to training
Fit has to use
Tit does, almost full until it crashes.. Will screen cap. on next attempt
Fit uses total 32.2 GB VRAM
Fit shouldnt crash it shows you have 40 GB
Fget a fresh config from the zip file
Fset only your training folder and model file paths
ZAlso helps to close all applications and disable hardware acceleration for browser
Zlet me reboot and do a stress test on the profile
LRank_3_T5_XXL_23500MB_11_35_Second_IT.json with V9 version on rtx3090, 8.94s/it
2024-10-25 20:14:29 INFO prepare CLIP-L for fp8: set to torch.float8_e4m3fn, set flux_train_network.py:509
embeddings to torch.bfloat16
INFO prepare T5XXL for fp8: set to torch.float8_e4m3fn, set embeddings flux_train_network.py:538
to torch.bfloat16, add hooks
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 50
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]2024-10-25 20:14:47 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1084
INFO text_encoder [0] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
INFO text_encoder [1] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
epoch 1/200
INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▏ | 34/10000 [05:04<24:45:40, 8.94s/it, avr_loss=0.417]T
2024-10-25 20:14:29 INFO prepare CLIP-L for fp8: set to torch.float8_e4m3fn, set flux_train_network.py:509
embeddings to torch.bfloat16
INFO prepare T5XXL for fp8: set to torch.float8_e4m3fn, set embeddings flux_train_network.py:538
to torch.bfloat16, add hooks
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 50
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]2024-10-25 20:14:47 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1084
INFO text_encoder [0] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
INFO text_encoder [1] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
epoch 1/200
INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▏ | 34/10000 [05:04<24:45:40, 8.94s/it, avr_loss=0.417]T
LYou can also temporarily enable the CUDA policy to use shared memory. And then calculate how much VRAM is missing because the CUDA error lies about the VRAM needed.
T
TIt actually goes up to 38.4/39.9GB just before it crashes.
Fah wait
Fhow much RAM you have?
Fphysical RAM
T64GB - it stays on first epoch step1. slowing climbing up to 38.4 and just as it s about to do first step. it gives cuda out of memory
Fok it should work
Fyou are using base FLUX model right?
F23.8 gb
Fbatch size 1 right?
Fit uses 32 gb VRAM
Fcombined
Tyes and batch 1 according to clean 8GB config
Fdid you set accelerate?
Tonly in the config.. but i noticed config shows fp16 instead ofbf16?
Fok it should be bf16
Fdoesnt work?
T
Fno not that one
Faccelerate
Fit is accurate
Fwell i dont know what else can be made need further debug. but your gpu also older gen i wonder if that makes difference
Fprobably best to rent a machine on massed compute
Tit was working before. I don't understand wht changed with the recent release 
Ftrain and use on your computer
Fyou can use older
Fjust make fresh install
Fdont run torch 2.5
Fit will stay as torch 2.4.1
Tit ran on torch 2.5 pre-release or beta..
Tis there a way to get that again?
Fah i see
Fit was release candidate
