wdym fine tuning on 16 GB but not lora?!
wdym fine tuning on 16 GB but not lora?!
Fye 
Fbecause of optimiziations
Dhow can fine tuning work on 16 GB when the model is already 12B * 2 bytes?
Fthere is half model training for lora but it is peaking like this
Fa lot of optimization with speed loss
FFthis above is lora training half model
Fit peaks 18 gb
JDo you have a config I can download for Flux training on Kohya, just to get setup quickly and I can make the changes for my computer specific information?
Dspeed loss?
Fyes slower training
Dah you mean loss of performance. was thinking of loss functions
Fi am preparing very best one
Xflux vs grok?
XWill grok get banned soon?
Ffirst flux likeliness 50 epoch not optimized yet
no face inpainting either
no face inpainting either
F
Fdont think so
Ffp16 accelerate vs bf16 accelerate the difference is humongous - early testing
Swhat I dont get, the ones on the left are still, same trained ohwx man but nothing like you, at all,. ? which makes fp16 unusable? I remember you had an article of this, but its out of my knowlege Doc!.
(https://medium.com/@furkangozukara/what-is-the-difference-between-fp16-and-bf16-here-a-good-explanation-for-you-d75ac7ec30fa)
(https://medium.com/@furkangozukara/what-is-the-difference-between-fp16-and-bf16-here-a-good-explanation-for-you-d75ac7ec30fa)
Medium
Updated and improved 29 July 2024
Fye according to fp16 it is supposed to work better
Fbut in sdxl it was also inferior
JShould I switch that lora config to bf16 instead of fp16? I'm getting OOM with 46GB
Fye BF16 better
Fi am getting results will update configs :d
Fwhere do you train on massed compute?
Jok.. I dont really understand this, but adjust maximum image sized from 2048 to 1644 allow me to get it to fit, but now only using 31GB
JRight now I'm on Runpod. I'll use massed about 25% of the time, but I haven't found a way to download the models I trained in any reasonable timeframe
Fupload hugging face :d
Fwhich i am doing
Fi am training at 1024x1024 no bucketing
F128 rank works fine
Fbf16 training not fp8
JI'll give that a try... Is there anyway to install runpodctl on massed instance?
Fyep you can do
Flooks like running model in FP16 precision instead of FP8 makes minimalistic difference
JI got runpodctl setup on massed computer. I'll see how my download speeds are for models. That was really my only complaint
Jon massed computer, when we're on the desktop, open the terminal, git check out, activate venv, then load kohya_gui.py?
JWhen i did that I got the old version, but when I go back to the terminal it says im up to date on the date on the sd3.flux branch
Fyou need to do this
Ffind kohya ss in apps
JI got it
Fopen a terminal
Fgit pull
Fah ok
Jthank you
Fyou are welcome
JI definitely think it's better to save off steps then epochs though, on that first runpod machine It's taking 1.5 hours to get through 5 epochs... needless to say, I changed that to steps on the massed instance
Fyou can do either way
Fsince i made repet 1 atm i save based on epochs