okay will do that. Thanks for all of your help.
okay will do that. Thanks for all of your help.
Flet me know results
Lwith version v9 I am getting 9.47 s/It (on a rtx 3090) I get some warnings like WARNING: Failed to find Windows SDK. is this normal? enable full bf16 training.
running training / 学習開始
num examples / サンプル数: 50
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]
epoch 1/200
2024-10-25 19:28:16 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▎ | 50/10000 [07:53<26:09:50, 9.47s/it, avr_loss=0.399]
epoch 2/200
running training / 学習開始
num examples / サンプル数: 50
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]
epoch 1/200
2024-10-25 19:28:16 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▎ | 50/10000 [07:53<26:09:50, 9.47s/it, avr_loss=0.399]
epoch 2/200
Fyes normal
Fspeed good
Factually speed better than even A6000 on linux with same config :d
Fvery good
LI also noticed a slight improvement in memory usage. 21gb of VRAM usage on average, I could try lowering the “Blocks to swap” setting to take advantage of all the VRAM. In 24GB_GPU_23150MB_10.2_second_it_Tier_1.json this value is at 10, I will try lowering it to 8.
Fit should speed up
Fevery block speeds up nice
Lalso try to test your LORA configurations. The Rank_1_29500MB_8_85_Second_IT.json is not ready to run on a 3090, right? It's the first time I try your LORA configurations, sorry if it's an obvious question.
LI run out of VRAM. I think I need about 29 gb of VRAM, and my 3090 falls short with its 24gb.
Fit is 30 gb
Fthe names tell your GB you need
Fyou need to rent cloud or purchase rtx 5090 - will be 32 gb very likely
Fi am waiting rtx 5090
Fi think its fine tuning speed will be like 2-3 second / it
LHere in Argentina it will be impossible to buy a 5090. But it is feasible to buy several 3090s because mining here is no longer a business and the prices of the 3090s have dropped a lot.
ZThe grid of all Grids.. 6048 images
Fthe speed on kaggle note bad
F4.45 second / it
Fgeneration
Z9 weights, 6 loras, 4 step variations, 28 prompts
Fand 2x gpu
Z4xA6000
F
Fnice way of doing it
Fi like swarm ui multi gpu back end
Z
Zyep, the grid feature is unbeatable
ZIt is really just for me to understand the process. You have been instrumental in making this happen @Furkan Gözükara SECourses ... I did the 140 Epochs B7 on 113 pics, then 60 Epochs on that model with same dataset
Ztesting the 2nd gen loras
Zfor fun 
T@Furkan Gözükara SECourses Unfortunately after clean install - orch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.90 GiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 8.51 GiB is allocated by PyTorch, and 1.70 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. What else can I try? I did Clean VS2022 17.8.14, Nvidia Driver 561.09 game ready, cuda 12.4, WindowsInstallStep1 installer option #1, option #2 cudun, Windows_Update_kohya_fix_flux_step2, Windows_install_torch 2.5. Loaded 8GB config from dreambooth
ZHow does your idle memory consumption look like?
Z
T
Zlet me fire up the config on my end
Fwatch the task manager
Fand see if it uses shared vram
Fduring trying to training
Fit has to use
Tit does, almost full until it crashes.. Will screen cap. on next attempt
Fit uses total 32.2 GB VRAM
Fit shouldnt crash it shows you have 40 GB
Fget a fresh config from the zip file
Fset only your training folder and model file paths
ZAlso helps to close all applications and disable hardware acceleration for browser
