Software Engineering Courses (SECourses)•2y ago

I had it for kohya, as I was using kohya until we noticed the quality isn't good anymore. so I thi

I had it for kohya, as I was using kohya until we noticed the quality isn't good anymore. so I think testing this for kohya only makes sense after you have found the perfect hyperparameters again. otherwise we'd compare against something that is bad for other reasons

Ddxqb I *had* it for kohya, as I was using kohya until we noticed the quality isn't go...

Furkan Gözükara SECourses•5/20/24, 10:54 PM

ye i found them

Furkan Gözükara SECourses•5/20/24, 10:54 PM

going to update now

dxqbOP•5/20/24, 11:04 PM

I DMed you a patch. The class it controls against is hardcoded to 'man', so if you are training a woman or something else, you need to change that

dxqbOP•5/20/24, 11:05 PM

parameters: leave everything as-is, but turn on TE2, too, and set both TE1 and TE2 learning rates to the main learning rate 1e-05

dxqbOP•5/20/24, 11:05 PM

if you want it for kohya, I'd have to first test it again tomorrow, as it was a while ago and I don't know if I have left it in a working state

dxqbOP•5/20/24, 11:07 PM

off to bed now, let me know if you have any questions!

Furkan Gözükara SECourses•5/21/24, 12:39 AM

Kohya config updated https://www.patreon.com/posts/very-best-for-of-89213064

AIGambino•5/21/24, 1:13 AM

Whats the difference between kohya and onetrainer?

AAIGambino Whats the difference between kohya and onetrainer?

Furkan Gözükara SECourses•5/21/24, 1:14 AM

different scripts

Furkan Gözükara SECourses•5/21/24, 1:14 AM

different features

Furkan Gözükara SECourses•5/21/24, 1:14 AM

and produces even different results in same settings :d

AIGambino•5/21/24, 1:14 AM

Interesting

AIGambino•5/21/24, 1:14 AM

How long should using 20 images take to train using the 24gb json file roughly?

AAIGambino How long should using 20 images take to train using the 24gb json file roughly?

Furkan Gözükara SECourses•5/21/24, 1:17 AM

depends on gpu

AIGambino•5/21/24, 1:17 AM

3090 is what I have

Furkan Gözükara SECourses•5/21/24, 1:17 AM

but lets say you did 150 repeat

Furkan Gözükara SECourses•5/21/24, 1:17 AM

ye around 1.7 second / it lets say

Furkan Gözükara SECourses•5/21/24, 1:17 AM

Furkan Gözükara SECourses•5/21/24, 1:18 AM

6000 * 1.7 second

AIGambino•5/21/24, 1:19 AM

in your latest post there what did you do change the 24gb json file configs?

AAIGambino in your latest post there what did you do change the 24gb json file configs?

Furkan Gözükara SECourses•5/21/24, 1:20 AM

learning rates

Furkan Gözükara SECourses•5/21/24, 1:20 AM

read the patreon post very detailed

Furkan Gözükara SECourses•5/21/24, 2:15 AM

i should test and see if SD 1.5 broken too or not :/

FFurkan Gözükara SECourses i should test and see if SD 1.5 broken too or not :/

dsienra•5/21/24, 2:23 AM

I'm training on an rtx 3090 with Gradient checkpointing off and CrossAttention sdpa at 1.52s/it

Ddsienra I'm training on an rtx 3090 with Gradient checkpointing off and CrossAttention s...

Furkan Gözükara SECourses•5/21/24, 2:31 AM

nice speed

Furkan Gözükara SECourses•5/21/24, 2:31 AM

spda speed up

Furkan Gözükara SECourses•5/21/24, 2:31 AM

but i dont know how much quality loss

dsienra•5/21/24, 2:33 AM

1.51s/it now, still slower than previous versions with the same settings but it is faster than using Gradient checkpointing on this capture is with an older version

FFurkan Gözükara SECourses nice speed

dsienra•5/21/24, 2:35 AM

I didn't know that this option can decrease quality, what about xformers, with it also can get quality loss?

dsienra•5/21/24, 2:37 AM

I get almost the same speeds with xformers and a little more vram usage 20gb or so

Ddsienra I didn't know that this option can decrease quality, what about xformers, with i...

Furkan Gözükara SECourses•5/21/24, 3:08 AM

yes in past both degraded

Furkan Gözükara SECourses•5/21/24, 3:08 AM

but it is getting better

dsienra•5/21/24, 3:17 AM

ok thanks!! I will check if I can see any difference

Furkan Gözükara SECourses•5/21/24, 3:18 AM

i will do a training with onetrainer

Furkan Gözükara SECourses•5/21/24, 3:18 AM

lets see difference between kohya

dsienra•5/21/24, 3:23 AM

that will be an interesting test!! a good test can be also to do a run test with xformers and spda, its a little speed up but can be very significant speed up for very large datasets and very long trainings.

Furkan Gözükara SECourses•5/21/24, 3:25 AM

doing a fresh install

Furkan Gözükara SECourses•5/21/24, 3:26 AM

of onetrainer

Furkan Gözükara SECourses•5/21/24, 3:26 AM

ye for massive dataset

Furkan Gözükara SECourses•5/21/24, 3:26 AM

you have to have xformers and batch size bigger

NNicB Yes. I'll work on getting that fixed tomorrow with the team

RMK•5/21/24, 3:27 AM

Hi just checking to see if this works now ? And if so, how do I do it?

dsienra•5/21/24, 3:31 AM

LR Scheduler sdpa never worked on kohya_ss an sdxl models, seems to be an interesting concept I read that it automatically adjusts the learning rate or something like that.

dsienra•5/21/24, 3:32 AM

I don't know if it was designed to work in conjunction with CrossAttention sdpa

Ddsienra LR Scheduler sdpa never worked on kohya_ss an sdxl models, seems to be an intere...

Furkan Gözükara SECourses•5/21/24, 3:36 AM

i use static LR

FFurkan Gözükara SECourses i use static LR

dsienra•5/21/24, 3:41 AM

Yes I know, I think is the best choice for a little dataset and for training just one subject, but what do you think for other cases like bigger and probably unbalanced datasets with multiple subjects? a smart lr could be nice.

Narva•5/21/24, 3:43 AM

I have trained a lora model in kohya and I have obtained better likeness and consistency than with a full dreambooth model

Narva•5/21/24, 3:43 AM

Lora is a underrated model

Ddsienra Yes I know, I think is the best choice for a little dataset and for training jus...

Furkan Gözükara SECourses•5/21/24, 3:44 AM

smart LR not working for stable diffusion

Furkan Gözükara SECourses•5/21/24, 3:44 AM

they work for LLMs

NNarva I have trained a lora model in kohya and I have obtained better likeness and con...

Furkan Gözükara SECourses•5/21/24, 3:44 AM

probably your config was not good for dreambooth. lora is scientifically proven to be inferior

I *had* it for kohya, as I was using kohya until we noticed the quality isn't good anymore. so I thi

I had it for kohya, as I was using kohya until we noticed the quality isn't good anymore. so I thi