<@205854764540362752> do you have experience with training multiple models at a time with Kohya on m
@Furkan Gözükara SECourses do you have experience with training multiple models at a time with Kohya on multi-GPU systems? I read what you wrote about the bug leading to excess memory utilization when multi-GPU finetuning, so you don't get linear increases in training speed with increases in GPU numbers.
I've been lucky enough to get a lot of credits from the great guys at TensorDock so I could try with multi-H100 SXM systems; I can confirm the bug but I can also tell you that you can run 2 instances of Kohya at the same time, and in that case you do get double the training speed.
Of course this is only useful if you have more than one model to train in the first place…
I've been lucky enough to get a lot of credits from the great guys at TensorDock so I could try with multi-H100 SXM systems; I can confirm the bug but I can also tell you that you can run 2 instances of Kohya at the same time, and in that case you do get double the training speed.
Of course this is only useful if you have more than one model to train in the first place…
![Spectral Analysis - [Short Concept Video for @thevoidz_ ]](https://cdn.answeroverflow.com/embeds/1320043031549972502/0/thumbnail.jpeg)
