Fi see
Fi think you should get speed between 50 second to 100 second it cant tell exact
F3060 gets 40 second / it
F
TI was getting 52 approx
F
Fyep sounds good
TAnything else i should do to make it clean.. just did a clean Nvidia studio driver install 565.90. Then will do Cuda, then will run your kohya scripts. Option #1, Option #2, Fix, Torch upgrade.
Fi think fresh nvidia driver with RESET settings
Fshould fix
Falso dont use studio driver
Fuse game ready
Fi am using it :d
FDriver Version: 561.09
Fmy driver
Tokay will do that. Thanks for all of your help.
Flet me know results
Lwith version v9 I am getting 9.47 s/It (on a rtx 3090) I get some warnings like WARNING: Failed to find Windows SDK. is this normal? enable full bf16 training.
running training / 学習開始
num examples / サンプル数: 50
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]
epoch 1/200
2024-10-25 19:28:16 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▎ | 50/10000 [07:53<26:09:50, 9.47s/it, avr_loss=0.399]
epoch 2/200
running training / 学習開始
num examples / サンプル数: 50
num batches per epoch / 1epochのバッチ数: 50
num epochs / epoch数: 200
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
steps: 0%| | 0/10000 [00:00<?, ?it/s]
epoch 1/200
2024-10-25 19:28:16 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715
D:\IA\kohya\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
steps: 0%|▎ | 50/10000 [07:53<26:09:50, 9.47s/it, avr_loss=0.399]
epoch 2/200
Fyes normal
Fspeed good
Factually speed better than even A6000 on linux with same config :d
Fvery good
LI also noticed a slight improvement in memory usage. 21gb of VRAM usage on average, I could try lowering the “Blocks to swap” setting to take advantage of all the VRAM. In 24GB_GPU_23150MB_10.2_second_it_Tier_1.json this value is at 10, I will try lowering it to 8.
Fit should speed up
Fevery block speeds up nice
Lalso try to test your LORA configurations. The Rank_1_29500MB_8_85_Second_IT.json is not ready to run on a 3090, right? It's the first time I try your LORA configurations, sorry if it's an obvious question.
LI run out of VRAM. I think I need about 29 gb of VRAM, and my 3090 falls short with its 24gb.
Fit is 30 gb
Fthe names tell your GB you need
Fyou need to rent cloud or purchase rtx 5090 - will be 32 gb very likely
Fi am waiting rtx 5090
Fi think its fine tuning speed will be like 2-3 second / it
LHere in Argentina it will be impossible to buy a 5090. But it is feasible to buy several 3090s because mining here is no longer a business and the prices of the 3090s have dropped a lot.
ZThe grid of all Grids.. 6048 images
Fthe speed on kaggle note bad
F4.45 second / it
Fgeneration
Z9 weights, 6 loras, 4 step variations, 28 prompts
Fand 2x gpu
Z4xA6000
F
Fnice way of doing it
Fi like swarm ui multi gpu back end
Z
Zyep, the grid feature is unbeatable
ZIt is really just for me to understand the process. You have been instrumental in making this happen @Furkan Gözükara SECourses ... I did the 140 Epochs B7 on 113 pics, then 60 Epochs on that model with same dataset
Ztesting the 2nd gen loras