I run out of VRAM. I think I need about 29 gb of VRAM, and my 3090 falls short with its 24gb.
I run out of VRAM. I think I need about 29 gb of VRAM, and my 3090 falls short with its 24gb.
Fit is 30 gb
Fthe names tell your GB you need
Fyou need to rent cloud or purchase rtx 5090 - will be 32 gb very likely
Fi am waiting rtx 5090
Fi think its fine tuning speed will be like 2-3 second / it
LHere in Argentina it will be impossible to buy a 5090. But it is feasible to buy several 3090s because mining here is no longer a business and the prices of the 3090s have dropped a lot.
ZThe grid of all Grids.. 6048 images
Fthe speed on kaggle note bad
F4.45 second / it
Fgeneration
Z9 weights, 6 loras, 4 step variations, 28 prompts
Fand 2x gpu
Z4xA6000
F
Fnice way of doing it
Fi like swarm ui multi gpu back end
Z
Zyep, the grid feature is unbeatable
ZIt is really just for me to understand the process. You have been instrumental in making this happen @Furkan Gözükara SECourses ... I did the 140 Epochs B7 on 113 pics, then 60 Epochs on that model with same dataset
Ztesting the 2nd gen loras
Zfor fun 
T@Furkan Gözükara SECourses Unfortunately after clean install - orch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.90 GiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 8.51 GiB is allocated by PyTorch, and 1.70 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. What else can I try? I did Clean VS2022 17.8.14, Nvidia Driver 561.09 game ready, cuda 12.4, WindowsInstallStep1 installer option #1, option #2 cudun, Windows_Update_kohya_fix_flux_step2, Windows_install_torch 2.5. Loaded 8GB config from dreambooth
ZHow does your idle memory consumption look like?
Z
T
Zlet me fire up the config on my end
Fwatch the task manager
Fand see if it uses shared vram
Fduring trying to training
Fit has to use
Tit does, almost full until it crashes.. Will screen cap. on next attempt
Fit uses total 32.2 GB VRAM
Fit shouldnt crash it shows you have 40 GB
Fget a fresh config from the zip file
Fset only your training folder and model file paths
ZAlso helps to close all applications and disable hardware acceleration for browser
Zlet me reboot and do a stress test on the profile
LRank_3_T5_XXL_23500MB_11_35_Second_IT.json with V9 version on rtx3090, 8.94s/it
2024-10-25 20:14:29 INFO prepare CLIP-L for fp8: set to torch.float8_e4m3fn, set flux_train_network.py:509
embeddings to torch.bfloat16
INFO prepare T5XXL for fp8: set to torch.float8_e4m3fn, set embeddings flux_train_network.py:538
to torch.bfloat16, add hooks
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 50
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]2024-10-25 20:14:47 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1084
INFO text_encoder [0] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
INFO text_encoder [1] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
epoch 1/200
INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▏ | 34/10000 [05:04<24:45:40, 8.94s/it, avr_loss=0.417]T
2024-10-25 20:14:29 INFO prepare CLIP-L for fp8: set to torch.float8_e4m3fn, set flux_train_network.py:509
embeddings to torch.bfloat16
INFO prepare T5XXL for fp8: set to torch.float8_e4m3fn, set embeddings flux_train_network.py:538
to torch.bfloat16, add hooks
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 50
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]2024-10-25 20:14:47 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1084
INFO text_encoder [0] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
INFO text_encoder [1] dtype: torch.float8_e4m3fn, device: cuda:0 train_network.py:1090
epoch 1/200
INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▏ | 34/10000 [05:04<24:45:40, 8.94s/it, avr_loss=0.417]T
LYou can also temporarily enable the CUDA policy to use shared memory. And then calculate how much VRAM is missing because the CUDA error lies about the VRAM needed.
T
TIt actually goes up to 38.4/39.9GB just before it crashes.
Fah wait
Fhow much RAM you have?
Fphysical RAM
T64GB - it stays on first epoch step1. slowing climbing up to 38.4 and just as it s about to do first step. it gives cuda out of memory
Fok it should work
Fyou are using base FLUX model right?