For multi-GPU training, does the set-up need 80gb vram TOTAL, or does EACH gpu need 80gb vram?
For multi-GPU training, does the set-up need 80gb vram TOTAL, or does EACH gpu need 80gb vram?
Ffor fine tuning
Feach gpu 80 gb
Ffor lora not necessary 48 gb gpus works great
GGottcha, thank you!
GOut of interest, what causes the VRAM per GPU requirements to jump up so much? Just curious on what's going on in the background processes that requires VRAM requirement to go from 24gb VRAM per GPU from single training to 80gb VRAM per GPU for muli-traning
Fextremely good question
Fi asked this to kohya
Fhe was also not sure
THow does multy GPU trainign work anyways?
Each GPU has a copy of a trained model and perform traning steps independantly, so how in the end we are not gettign two separate models?
Each GPU has a copy of a trained model and perform traning steps independantly, so how in the end we are not gettign two separate models?
Tthey cannot possibly synchronize model state after each step, that woudl take ages (minutes)
TAnd if the models are just merged as last step - would that mean the final model can have questionable quality (missed local minimas and such)?
Fwith diffuser models
Feach models loaded fully into vram
Flike flux sd etc
Fso we have exact replicate
Tyeah, but how the state of each model loaded into GPU sync up?
TSo the training result in single model, not multiple models
Fthey are snyched at steps
Fso it is being increased batch size
Nlong time no see. I had trained few flux loras earlier, and had to reinstall my windows recently. After that I had to reinstall kohya, and now can't train anymore. First it didn't create venv (idk, maybe it's because latest installer or something else). I've installed kohya from original repo, and it started to work. But now I have another problem. Training speed reduced. Much. ~500s/it. I guess it's because it's using ram or something. Earlier I noticed it's caching t5 and clip-l to ram (to free vram), now there is no such notice. Any suggestions?
Ngpu 4060ti 16gb, ram 64 gb
N
NI'm using the same configs I've used earlier
Nit's gpu fallback turned on, because when I'm turning it off, I'm getting out of memory error
Fwhat is your gpu and which config you are using?
Fgive me exact name pls
Ngpu 4060ti 16gb, ram 64 gb
Fok use this config 12GB_GPU_11100MB_9.5_second_it_Tier_1.json
Falso install latest nvidia but make a clean installation so all configs reset
Nnvidia driver?
Nit's dreambooth training or lora tab?
Fthis is dreambooth
Fyes nvidia driver
Ffor lora you can use either of these
F12GB_GPU_Quality_Tier_4_FP8_11050MB_5.5_Second_IT.json
F16GB_GPU_Quality_Tier_3_14700MB_9.2_Second_IT.json
F16GB_GPU_Quality_Tier_2_14800MB_13.4_Second_IT_and_T5_Attention.json
Nthat's my config I've used earlier and it worked fine with speed ~16s/it
Fwell kohya made a lot of updates so our updated configs working
Fcant tell for older configs
Nyep, I guess it's because I've used v19 of your installer and it worked fine for me)
Fye it is old.
Fwe updated after re-testing after all changes of kohya
Nhope changes were useful)
Fyep they work great
Nyep, used your 16GB_GPU_Quality_Tier_2_14800MB_13.4_Second_IT_and_T5_Attention.json config and speed is back to previous (even bit faster). Thanks! Tried to start dreambooth training with ok use this config 12GB_GPU_11100MB_9.5_second_it_Tier_1.json and got 20+ s/it. Maybe it's first few epochs.
Fwait it will get faster
Fwait like 50 100 steps to see actual speed
N
Tbtw, I noticed that kohya calculates step speed incorrectly, including intermediate checkpoint save times.
So each time it saves calculation getting more and more off
So each time it saves calculation getting more and more off