Basically, you get 1.68 s/it on 1 and 2.36 s/it on the 2x system [fixed it/s → s/it]
Basically, you get 1.68 s/it on 1 and 2.36 s/it on the 2x system [fixed it/s → s/it]
MCorrect me if I'm wrong but this is better-than-linear scaling, right? I mean, it takes far less than 2x 1.68s (3.36 s) to do a step
MBTW I don't know if it happens to you too, but Kohya_ss ends training with a bunch of errors even when everything went fine… Pretty scary…
Fno it didnt scale linearly
Fare the pci ex?
Fyou got like only 30% speed up
MI noticed VRAM load is disproportionately higher when running the finetune on the 2x GPU: 100% on both GPUs vs. 45% on the single system. [CPU → GPU]
MThat is a lot of overhead, possibly a bug in the script?
Fi reported this already
Fkohya doesnt know
Ffrom 28 gb to 79 gb it goes
MAh yes now I remember!
-My error remains.... https://www.youtube.com/watch?v=AY6DMBCIZ3A&t=400s ... I tried 19 times, but not easy to catch up.. sorry to my knoledge and my brain bad..... : ( and bad my english...
YouTubeSECourses
Updated for SDXL 1.0. How to install #Kohya SS GUI trainer and do #LoRA training with Stable Diffusion XL (#SDXL) this is the video you are looking for. I have shown how to install Kohya from scratch. The best parameters to do LoRA training with SDXL. How to use Kohya SDXL LoRAs with ComfyUI. How to do checkpoint comparison with SDXL LoRAs and m...
-message.txt3.44KB
Flooks like kohya install error
Fyou can use my this installer
Fit works with sdxl too
FPatreon
Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon
FYouTubeSECourses
If you are interested in using AI, generative AI applications, open source applications in your computer, than this is the most fundamental and important tutorial that you need. In this tutorial I show and explain how to properly install appropriate Python versions accurately, how to switch between different Python versions, how to install diffe...
Fif you need extra help i am giving private consultation
Y@Furkan Gözükara SECourses hi again, in the dreambooth training, why you not training the TE? bad results?
Fit doesnt work sadly
Fthey didnt add yet 
Yah damn ohwell my captions is natural language anyway so
Ythank u!
Fyou are welcome
Fsoon i will publish a video about improved clip l hopefully 
Fit will improve quality withno changes
Ywait if TE is not trainable, I don't need to caption anymore?
Fno you need to caption
Fcaptions are encoded and cached and used
U
�Hey @Furkan Gözükara SECourses . Been trying training in Runpod, I went with the pytorch 2.2 as explained in runpod installation in kohya v16 script of yours. Twice in a row the training has stopped, first it saved epoch 10, then stopped at epoch 20. i started new training using my saved epoch 10 checkpoint, now it proceeded to epoch 50, then I got this same error: (weird, because I selected 400 GB harddisk):
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"
�The last file was only saved to 21 GB, so it didn't complete for some reason
Flooks to me like pod hdd failed
�happened again: maybe turn off memory efficient save?
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
Fyep turn it off
Fif you have sufficient ram
Fshouldnt be needed
�running the a40 GPU
�oh, you're talking normal RAM.
�
�says 50 GB RAM, that's a weird amount
�ok, disabled memory efficient save now, continuing from last created epoch
Fyes system ram
MIf I may, be careful how many times you save intermediate [models, not modules] and states; I ended up with a full HD very soon. Remember that when Kohya save a training state (those folders whose name ends with 'state'), there's also a partially-trained, but full-size model in it. If it's a Flux-dev finetune, each file is 21 GB IIRC…
TDo you mean that if I need to resume, so I have to uncheck "memory efficient save" and set "Save last N epochs state" to 2 (if I want to keep the last 2 epoch ? (I don't really understand this setting, if I keep 200 epoch, save every 25 epoch and save last 2 epoch, is that mean that every epoch, the state is saved but keeping only 2 last ? or it keep only the epoch number 175 and 200 (state) ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?
�instead of setting flux dev as base, you set i.e. mycheckpoint-000075.safetensors. I would now rename the model to mycheckpoint-from75 so you remember where you started, then rename the numbers after by adding 75
Tin this case, do you need to set also "save state with checkpoint" ?
�everything else is the same as before
�@Furkan Gözükara SECourses Is it possible to have 1.jpg 1.txt 2.jpg 2.txt while the rest of the pics are not manually captioned?