Hey @Dr. Furkan Gözükara how’s it going with your text encoder sdxl training tests?
Hey @Dr. Furkan Gözükara how’s it going with your text encoder sdxl training tests?
RBtw, kohya finally merged passing args to wand PR. It’s now available on the dev branch. It’s working now on kohya_ss by passing command in cli with “—log_config”
Frunning tests right now
F
RGreat looking forward to it. I’m testing style model but want to go banck to testing subject models with Adafactor vs adamW8bit vs prodigy
Rnice but?
V@Dr. Furkan Gözükara Hey Furkan which is the best one to upscale an image
H
AHow can I fix it? It seemed to work last week, no idea what happened since then?
AI tried this way through the unified loader faceid but still no luck
A
Ftesting x/y/z
Dnext time you are running a OneTrainer comparison, @Dr. Furkan Gözükara , would you mind adding 1 run with a change that I have made to OneTrainer? it's just a couple of lines of code, I'd send you a patch file.
it allows you to set TE1 and TE2 (!) to full learning rate, same as unet, without overcooking it.
I think it improves quality. but it's subtle and I'd like a second opinion
it allows you to set TE1 and TE2 (!) to full learning rate, same as unet, without overcooking it.
I think it improves quality. but it's subtle and I'd like a second opinion
Dor kohya, I have it for kohya, too. just didn't use it now because of the learning rate issue.
Fwhat do you mean by that?
Fcan you tell me more details
Dsure. in my opinion, the TEs are overcooking because we let them just run in one direction, uncontrolled. When you train the unet, there is loss function - which is how good it is denoising an image. when you enable TE training, it simply also optimizes the TE parameters, but the loss function is still only how good the denoising of unet is. so you allow the TE to optimize towards your training data set and it looses all flexibility.
Dso I have added a loss function that also controls the TE.
DI hope that was kinda clear? difficult to explain in two sentences
Fdid you make a PR for this?
Fonly for one trainer?
Dno, I've talked to the OT developer and he is not interested. he thinks it is too specific for my/our use case. he also thinks the same can be done with additional embeddings - but I've tried that. it's worse.
AThe workflow works on this clip but other clips just dont load at all and give that error
Dso whenever you have the chance, it would be great to get your opinion.
Fsure i would like to test
Fcurrently i was using kohya
Fdo you have same for kohya?
Fthat would be a nice comparison but i can test on OT too
DI had it for kohya, as I was using kohya until we noticed the quality isn't good anymore. so I think testing this for kohya only makes sense after you have found the perfect hyperparameters again. otherwise we'd compare against something that is bad for other reasons
Fye i found them
Fgoing to update now
DI DMed you a patch. The class it controls against is hardcoded to 'man', so if you are training a woman or something else, you need to change that
Dparameters: leave everything as-is, but turn on TE2, too, and set both TE1 and TE2 learning rates to the main learning rate 1e-05
Dif you want it for kohya, I'd have to first test it again tomorrow, as it was a while ago and I don't know if I have left it in a working state
Doff to bed now, let me know if you have any questions!
FKohya config updated https://www.patreon.com/posts/very-best-for-of-89213064
AWhats the difference between kohya and onetrainer?
Fdifferent scripts
Fdifferent features
Fand produces even different results in same settings :d
AInteresting
AHow long should using 20 images take to train using the 24gb json file roughly?
Fdepends on gpu
A3090 is what I have
Fbut lets say you did 150 repeat
Fye around 1.7 second / it lets say
Fso
F6000 * 1.7 second
Ain your latest post there what did you do change the 24gb json file configs?
Flearning rates
Fread the patreon post very detailed