thanks!! I hope so!! and I hope that 24gb vram will be enough to train flux + clip_l
thanks!! I hope so!! and I hope that 24gb vram will be enough to train flux + clip_l
Fcurrently only unet trained
Fwith kohya lora
F24 gb barely sufficient
Fupdated post with experiments
DI will check it
F
Iwith 24GB only Loras?
FYe this is only lora but I think fine tuning has better optimizations we will see. Also I asked kohya since there is half model training but still uses 18 gb gpu
Fmore than 50 epoch completed time to start doing some testing
Dso nothing for 16 GB?
Fthere is one config supposed to work but not working i messaged him
Fi think fine tuning may work for 16
Ffor lora i think there is bug
F32 rank full precision lora is 613 mb
Dwdym fine tuning on 16 GB but not lora?!
Fye 
Fbecause of optimiziations
Dhow can fine tuning work on 16 GB when the model is already 12B * 2 bytes?
Fthere is half model training for lora but it is peaking like this
Fa lot of optimization with speed loss
FFthis above is lora training half model
Fit peaks 18 gb
JDo you have a config I can download for Flux training on Kohya, just to get setup quickly and I can make the changes for my computer specific information?
Dspeed loss?
Fyes slower training
Dah you mean loss of performance. was thinking of loss functions
Fi am preparing very best one
Xflux vs grok?
XWill grok get banned soon?
Ffirst flux likeliness 50 epoch not optimized yet
no face inpainting either
no face inpainting either
F
Fdont think so
Ffp16 accelerate vs bf16 accelerate the difference is humongous - early testing
Swhat I dont get, the ones on the left are still, same trained ohwx man but nothing like you, at all,. ? which makes fp16 unusable? I remember you had an article of this, but its out of my knowlege Doc!.
(https://medium.com/@furkangozukara/what-is-the-difference-between-fp16-and-bf16-here-a-good-explanation-for-you-d75ac7ec30fa)
(https://medium.com/@furkangozukara/what-is-the-difference-between-fp16-and-bf16-here-a-good-explanation-for-you-d75ac7ec30fa)
Medium
Updated and improved 29 July 2024
Fye according to fp16 it is supposed to work better
Fbut in sdxl it was also inferior
JShould I switch that lora config to bf16 instead of fp16? I'm getting OOM with 46GB
Fye BF16 better
Fi am getting results will update configs :d
Fwhere do you train on massed compute?
Jok.. I dont really understand this, but adjust maximum image sized from 2048 to 1644 allow me to get it to fit, but now only using 31GB
JRight now I'm on Runpod. I'll use massed about 25% of the time, but I haven't found a way to download the models I trained in any reasonable timeframe
Fupload hugging face :d
Fwhich i am doing
Fi am training at 1024x1024 no bucketing
F128 rank works fine
Fbf16 training not fp8
JI'll give that a try... Is there anyway to install runpodctl on massed instance?
