Software Engineering Courses (SECourses)•16mo ago

https://x.com/Lykon4072/status/1834917552209744318

@linaqruf_ @oron1208 Wait for 8b. It's basically Flux without distillation and heavy hands dpo. This should make it easy to finetune (and dpo).
We're also trying a new scaling down mechanism for mmdit, the new 2b is gonna work much better.

Twitter

•

9/14/24, 11:30 AM

Furkan Gözükara SECourses•9/15/24, 2:20 AM

full fine tuning

MMaxivi https://x.com/Lykon4072/status/1834917552209744318

Furkan Gözükara SECourses•9/15/24, 2:20 AM

nice

Furkan Gözükara SECourses•9/15/24, 2:20 AM

it is also fp16 atm

Furkan Gözükara SECourses•9/15/24, 2:20 AM

not even fp8 it will be lower

FFurkan Gözükara SECourses full fine tuning

MaxiviOP•9/15/24, 2:22 AM

you can try training on a a100 to see if results are good first

MMaxivi you can try training on a a100 to see if results are good first

Furkan Gözükara SECourses•9/15/24, 2:28 AM

not needed

Furkan Gözükara SECourses•9/15/24, 2:29 AM

it works perfect on A6000

Furkan Gözükara SECourses•9/15/24, 2:29 AM

fused back pass reduced VRAM usage by far

Furkan Gözükara SECourses•9/15/24, 2:29 AM

also there are block swapping

Furkan Gözükara SECourses•9/15/24, 2:29 AM

so it will work on 24gb too

Furkan Gözükara SECourses•9/15/24, 2:29 AM

but fp8 didnt make any diff

MaxiviOP•9/15/24, 2:29 AM

but very slow though right?

Furkan Gözükara SECourses•9/15/24, 2:29 AM

for some reason

MMaxivi but very slow though right?

Furkan Gözükara SECourses•9/15/24, 2:29 AM

same speed as lora

MaxiviOP•9/15/24, 2:29 AM

with block swapping?

MMaxivi with block swapping?

Furkan Gözükara SECourses•9/15/24, 2:29 AM

no without

Furkan Gözükara SECourses•9/15/24, 2:29 AM

with block it will be slower i expect

Furkan Gözükara SECourses•9/15/24, 2:30 AM

MaxiviOP•9/15/24, 2:30 AM

but it wont fit on 24gb without block swapping i think

MMaxivi but it wont fit on 24gb without block swapping i think

Furkan Gözükara SECourses•9/15/24, 2:33 AM

with fp8 it should

Furkan Gözükara SECourses•9/15/24, 2:33 AM

but it didnt work

MaxiviOP•9/15/24, 2:33 AM

yeah the fp8 models arent making any difference not sure why

MMaxivi yeah the fp8 models arent making any difference not sure why

Furkan Gözükara SECourses•9/15/24, 2:36 AM

just reported it as a bug to the kohya

Furkan Gözükara SECourses•9/15/24, 2:36 AM

it should reduce vram to fit into 24 gb

Furkan Gözükara SECourses•9/15/24, 2:37 AM

cpu offloading almost there

FFurkan Gözükara SECourses just reported it as a bug to the kohya

MaxiviOP•9/15/24, 4:02 AM

what kohya said

If you specify fp8_base for LoRA training, the flux will be cast to fp8 from bf16, so the VRAM usage will be the same even with full (bf16) base model.**

FFurkan Gözükara SECourses fused back pass reduced VRAM usage by far

dxqb•9/15/24, 7:55 AM

kohya now has fused back pass?

dxqb•9/15/24, 10:21 AM

what did you have for Lora? I'm getting 5.7 s/it using OT, bs1 adafactor fused backpass

FluffyQuack•9/15/24, 11:30 AM

Results from trying to train a character + a puppet mascot character

20240915_1328_Oil_painting_of_a_MSFio_woman_with_orang_flux1-dev_2084106653.png

MMaxivi what kohya said If you specify fp8_base for LoRA training, the flux will be cas...

Furkan Gözükara SECourses•9/15/24, 11:36 AM

he said fp8 doesnt work

Ddxqb kohya now has fused back pass?

Furkan Gözükara SECourses•9/15/24, 11:36 AM

for fine tuning yes

Furkan Gözükara SECourses•9/15/24, 11:36 AM

works amazing

Furkan Gözükara SECourses•9/15/24, 11:44 AM

looks like my checkpoints totally cooked

Furkan Gözükara SECourses•9/15/24, 11:46 AM

Furkan Gözükara SECourses•9/15/24, 11:47 AM

i guess i need to lower LR a lot

Dazzastrous [4090]•9/15/24, 11:47 AM

getting shredded

FluffyQuack•9/15/24, 11:53 AM

Has anyone done tests regarding interactions between multiple LoRAs, or is there documentation somewhere how it works in detail (assuming it doesn't get so technical it goes over my head)

For instance, does LoRA load order matter? Is it better to train a model with two characters (like the one I did above with a character + mascot) compared to training two different LoRAs and using them both at the same time?

FFluffyQuack Has anyone done tests regarding interactions between multiple LoRAs, or is there...

Furkan Gözükara SECourses•9/15/24, 12:08 PM

it doenst work well

Furkan Gözükara SECourses•9/15/24, 12:08 PM

you have to train both in same image usually

FFluffyQuack Results from trying to train a character + a puppet mascot character

Aikage•9/15/24, 2:31 PM

Metal slug

atlas170•9/15/24, 2:35 PM

Is there any concensus on what is causing the grid lines/scan lines in some Flux LoRA's? I've read anything from: VAE issue, CUDA version issue, a bug in kohya, anti-AI watermark in training dataset. Has anyone successfully narrowed down what was causing it?

dxqb•9/15/24, 3:06 PM

I don't see anything here, but I've seen grid lines when I've tried to use a Lora at very high weights, like 2

Aatlas170 Is there any concensus on what is causing the grid lines/scan lines in some Flux...

Furkan Gözükara SECourses•9/15/24, 3:18 PM

it happens to me when i generate high res images

virginia•9/15/24, 3:29 PM

@Dr. Furkan Gözükara good afternoon, as to your liveportrait, could I put a video and image in a ratio 3:4, and get generated video also in 3:4?

druhl•9/15/24, 3:38 PM

There was so much fuss about the GGUF models/ people running out of VRAM/ Flux being too large, that I spent the entire day today with my home PC (12GB 4070 Super) & ComfyUI, trying various models. To my surprise, I found:

1) ComfyUI could load and generate with the original 23GB Flux Dev Model (reasonable speed) on my PC & FP8 T5XXL. I do not understand how people are running out of VRAM? It seems both swarm UI and comfy ui have some resource management going on inside which lets you use the dev model. Am I missing something? Since it was working, I loaded Florence2 and SAM2 as well, ended up generating + inpainting with relative ease.

2) As for the GGUF models, I actually found that they took much, much longer than the 23GB model to generate images!! I tried Q6_K and Q8_O and was able to do the above operations with them as well. The quality was good, but obviously not better than the full size dev model.

3) I found that Swarm UI is somehow doing a much better job at generating with the LoRAs+Img2Img. On the other hand, it seems SwarmUI can't work with Controlnets+LoRAs at all. The results are laughable tbh. The inpainting on ComfyUI is the best!

TitanTaliwacker•9/15/24, 5:21 PM

So I am pulling my hair out (and I am already bald). I really appreciate all the work put in to the tutorials and I have followed each step. I am now getting an error that shows "13:11:50-486153 ERROR LoRA type must be set to 'Flux1' or 'Flux1 OFT' if Flux1 checkbox is checked." Flux is auto checked as the model is in there and loaded. Please let me know what I can do to fix this.

TitanTaliwacker•9/15/24, 5:24 PM

nvm! I think I got it.

Vvirginia @Dr. Furkan Gözükara good afternoon, as to your liveportrait, could I put a vide...

Furkan Gözükara SECourses•9/15/24, 6:01 PM

yes it should work

Ddruhl There was so much fuss about the GGUF models/ people running out of VRAM/ Flux b...

Furkan Gözükara SECourses•9/15/24, 6:01 PM

1 , 2 , 3 are all correct :d

TTitanTaliwacker nvm! I think I got it.

Furkan Gözükara SECourses•9/15/24, 6:02 PM

did you solve, are you using our config auto installer?

https://x.com/Lykon4072/status/1834917552209744318

Similar Threads