time to do some fine tuning
time to do some fine tuning
Mjust got word that sd3 8b will be out soon, no distilation and easier to train and finetune
Flicence?
Mi think same as the the updated sd3 medium one with more clarity
Mlykon mentioned
Mlooks like it can be better than flux
Fnice
M@linaqruf_ @oron1208 Wait for 8b. It's basically Flux without distillation and heavy hands dpo. This should make it easy to finetune (and dpo).
We're also trying a new scaling down mechanism for mmdit, the new 2b is gonna work much better.
We're also trying a new scaling down mechanism for mmdit, the new 2b is gonna work much better.
9/14/24, 11:30 AM
Ffull fine tuning
Fnice
Fit is also fp16 atm
Fnot even fp8 it will be lower
Myou can try training on a a100 to see if results are good first
Fnot needed
Fit works perfect on A6000
Ffused back pass reduced VRAM usage by far
Falso there are block swapping
Fso it will work on 24gb too
Fbut fp8 didnt make any diff
Mbut very slow though right?
Ffor some reason
Fsame speed as lora
Mwith block swapping?
Fno without
Fwith block it will be slower i expect
F
Mbut it wont fit on 24gb without block swapping i think
Fwith fp8 it should
Fbut it didnt work 
Myeah the fp8 models arent making any difference not sure why
Fjust reported it as a bug to the kohya
Fit should reduce vram to fit into 24 gb
Fcpu offloading almost there
Mwhat kohya said
If you specify fp8_base for LoRA training, the flux will be cast to fp8 from bf16, so the VRAM usage will be the same even with full (bf16) base model.**
If you specify fp8_base for LoRA training, the flux will be cast to fp8 from bf16, so the VRAM usage will be the same even with full (bf16) base model.**
Dkohya now has fused back pass?
Dwhat did you have for Lora? I'm getting 5.7 s/it using OT, bs1 adafactor fused backpass
FResults from trying to train a character + a puppet mascot character
Fhe said fp8 doesnt work
Ffor fine tuning yes
Fworks amazing
Flooks like my checkpoints totally cooked
F
Fi guess i need to lower LR a lot
![Dazzastrous [4090]](https://cdn.discordapp.com/avatars/871389071686135838/c4a5f4d81d9eeb18d856295d692e2c07.webp?size=40)
Dgetting shredded
FHas anyone done tests regarding interactions between multiple LoRAs, or is there documentation somewhere how it works in detail (assuming it doesn't get so technical it goes over my head)
For instance, does LoRA load order matter? Is it better to train a model with two characters (like the one I did above with a character + mascot) compared to training two different LoRAs and using them both at the same time?
For instance, does LoRA load order matter? Is it better to train a model with two characters (like the one I did above with a character + mascot) compared to training two different LoRAs and using them both at the same time?
Fit doenst work well
Fyou have to train both in same image usually
AMetal slug 
AIs there any concensus on what is causing the grid lines/scan lines in some Flux LoRA's? I've read anything from: VAE issue, CUDA version issue, a bug in kohya, anti-AI watermark in training dataset. Has anyone successfully narrowed down what was causing it?
DI don't see anything here, but I've seen grid lines when I've tried to use a Lora at very high weights, like 2
Fit happens to me when i generate high res images