Software Engineering Courses (SECourses)•11mo ago

How does multy GPU trainign work anyways? Each GPU has a copy of a trained model and perform traning

How does multy GPU trainign work anyways?
Each GPU has a copy of a trained model and perform traning steps independantly, so how in the end we are not gettign two separate models?

TimsonOP•2/22/25, 11:52 PM

they cannot possibly synchronize model state after each step, that woudl take ages (minutes)

TimsonOP•2/22/25, 11:55 PM

And if the models are just merged as last step - would that mean the final model can have questionable quality (missed local minimas and such)?

TTimson How does multy GPU trainign work anyways? Each GPU has a copy of a trained model...

Furkan Gözükara SECourses•2/23/25, 10:01 AM

with diffuser models

Furkan Gözükara SECourses•2/23/25, 10:01 AM

each models loaded fully into vram

Furkan Gözükara SECourses•2/23/25, 10:02 AM

like flux sd etc

Furkan Gözükara SECourses•2/23/25, 10:02 AM

so we have exact replicate

FFurkan Gözükara SECourses each models loaded fully into vram

TimsonOP•2/23/25, 10:41 AM

yeah, but how the state of each model loaded into GPU sync up?

TimsonOP•2/23/25, 10:41 AM

So the training result in single model, not multiple models

TTimson yeah, but how the state of each model loaded into GPU sync up?

Furkan Gözükara SECourses•2/23/25, 10:45 AM

they are snyched at steps

Furkan Gözükara SECourses•2/23/25, 10:45 AM

so it is being increased batch size

novsky•2/23/25, 4:31 PM

long time no see. I had trained few flux loras earlier, and had to reinstall my windows recently. After that I had to reinstall kohya, and now can't train anymore. First it didn't create venv (idk, maybe it's because latest installer or something else). I've installed kohya from original repo, and it started to work. But now I have another problem. Training speed reduced. Much. ~500s/it. I guess it's because it's using ram or something. Earlier I noticed it's caching t5 and clip-l to ram (to free vram), now there is no such notice. Any suggestions?

novsky•2/23/25, 4:32 PM

gpu 4060ti 16gb, ram 64 gb

novsky•2/23/25, 4:32 PM

novsky•2/23/25, 4:33 PM

I'm using the same configs I've used earlier

novsky•2/23/25, 4:39 PM

it's gpu fallback turned on, because when I'm turning it off, I'm getting out of memory error

Nnovsky long time no see. I had trained few flux loras earlier, and had to reinstall my ...

Furkan Gözükara SECourses•2/23/25, 4:53 PM

what is your gpu and which config you are using?

Furkan Gözükara SECourses•2/23/25, 4:53 PM

give me exact name pls

novsky•2/23/25, 4:54 PM

gpu 4060ti 16gb, ram 64 gb

Nnovsky gpu 4060ti 16gb, ram 64 gb

Furkan Gözükara SECourses•2/23/25, 4:54 PM

ok use this config 12GB_GPU_11100MB_9.5_second_it_Tier_1.json

Furkan Gözükara SECourses•2/23/25, 4:54 PM

also install latest nvidia but make a clean installation so all configs reset

FFurkan Gözükara SECourses also install latest nvidia but make a clean installation so all configs reset

novsky•2/23/25, 4:56 PM

nvidia driver?

FFurkan Gözükara SECourses ok use this config 12GB_GPU_11100MB_9.5_second_it_Tier_1.json

novsky•2/23/25, 4:58 PM

it's dreambooth training or lora tab?

Nnovsky it's dreambooth training or lora tab?

Furkan Gözükara SECourses•2/23/25, 4:59 PM

this is dreambooth

Nnovsky nvidia driver?

Furkan Gözükara SECourses•2/23/25, 4:59 PM

yes nvidia driver

Furkan Gözükara SECourses•2/23/25, 5:00 PM

for lora you can use either of these

Furkan Gözükara SECourses•2/23/25, 5:00 PM

12GB_GPU_Quality_Tier_4_FP8_11050MB_5.5_Second_IT.json

Furkan Gözükara SECourses•2/23/25, 5:00 PM

16GB_GPU_Quality_Tier_3_14700MB_9.2_Second_IT.json

Furkan Gözükara SECourses•2/23/25, 5:00 PM

16GB_GPU_Quality_Tier_2_14800MB_13.4_Second_IT_and_T5_Attention.json

novsky•2/23/25, 5:11 PM

that's my config I've used earlier and it worked fine with speed ~16s/it

LoRa-1024-att-mask-200ep-3000.json6.48KB

Nnovsky that's my config I've used earlier and it worked fine with speed ~16s/it

Furkan Gözükara SECourses•2/23/25, 5:13 PM

well kohya made a lot of updates so our updated configs working

Furkan Gözükara SECourses•2/23/25, 5:13 PM

cant tell for older configs

FFurkan Gözükara SECourses well kohya made a lot of updates so our updated configs working

novsky•2/23/25, 5:14 PM

yep, I guess it's because I've used v19 of your installer and it worked fine for me)

Nnovsky yep, I guess it's because I've used v19 of your installer and it worked fine for...

Furkan Gözükara SECourses•2/23/25, 5:15 PM

ye it is old.

Furkan Gözükara SECourses•2/23/25, 5:16 PM

we updated after re-testing after all changes of kohya

FFurkan Gözükara SECourses we updated after re-testing after all changes of kohya

novsky•2/23/25, 5:19 PM

hope changes were useful)

Furkan Gözükara SECourses•2/23/25, 5:22 PM

yep they work great

novsky•2/23/25, 5:25 PM

yep, used your 16GB_GPU_Quality_Tier_2_14800MB_13.4_Second_IT_and_T5_Attention.json config and speed is back to previous (even bit faster). Thanks! Tried to start dreambooth training with ok use this config 12GB_GPU_11100MB_9.5_second_it_Tier_1.json and got 20+ s/it. Maybe it's first few epochs.

Nnovsky yep, used your 16GB_GPU_Quality_Tier_2_14800MB_13.4_Second_IT_and_T5_Attention.j...

Furkan Gözükara SECourses•2/23/25, 5:26 PM

wait it will get faster

Furkan Gözükara SECourses•2/23/25, 5:26 PM

wait like 50 100 steps to see actual speed

novsky•2/23/25, 5:28 PM

@Furkan Gözükara SECourses thanks again for assist

FFurkan Gözükara SECourses wait like 50 100 steps to see actual speed

TimsonOP•2/23/25, 11:10 PM

btw, I noticed that kohya calculates step speed incorrectly, including intermediate checkpoint save times.
So each time it saves calculation getting more and more off

TimsonOP•2/23/25, 11:11 PM

lora save time is very fast, but for full checkpoint around 55 seconds on massed compute

TTimson lora save time is very fast, but for full checkpoint around 55 seconds on massed...