Software Engineering Courses (SECourses)•15mo ago

@hamuz I am using this one and enjoy the results. https://medium.com/@zhiwangshi28/why-flux-lora-so

@hamuz I am using this one and enjoy the results. https://medium.com/@zhiwangshi28/why-flux-lora-so-hard-to-train-and-how-to-overcome-it-a0c70bc59eaf

OinkSauce•10/9/24, 5:28 PM

It seems like, for now, if you are only training one subject, the dev model is probably your go to.

If you want to try something fancy, creating a full finetune checkpoint then extracting the LORA has given me my best LORA's

Wwardensc2 just learn to use flux dev to train first dont need to use the de-distilled mode...

hamuz•10/9/24, 5:28 PM

Alright, thanks man! I’ll train a few different LoRAs with different settings, and I’ll share my experiences here with you. Maybe after doing enough research and gaining some knowledge, I’ll look into the distilled and de-distilled versions

VVigilence @hamuz I am using this one and enjoy the results. https://medium.com/@zhiwangsh...

wardensc2•10/9/24, 5:28 PM

this one I dont see anything different compare with Flux dev Lora

wardensc2•10/9/24, 5:29 PM

just create different image only

wardensc2•10/9/24, 5:29 PM

People also say the same

Wwardensc2 for Flux Dev model, the best setting we already have, just use @Dr. Furkan Gözük...

hamuz•10/9/24, 5:29 PM

ofcourse I will

VigilenceOP•10/9/24, 5:29 PM

@wardensc2 You tried both? Which output was more aligned to what you used for data?

VVigilence @wardensc2 You tried both? Which output was more aligned to what you used for d...

wardensc2•10/9/24, 5:29 PM

yes I try both

wardensc2•10/9/24, 5:30 PM

compare 2 loras: Flux and that model

wardensc2•10/9/24, 5:30 PM

just different image only

wardensc2•10/9/24, 5:30 PM

nothing improve

wardensc2•10/9/24, 5:30 PM

people suspect it just some kind of merge checkpoint and pretend something new

hamuz•10/9/24, 5:31 PM

Also, has anyone here rented and used an H100? Is there really a significant speed increase in training? Compared to 4x A100

VigilenceOP•10/9/24, 5:31 PM

There is also the method that ai toolkit uses to train. Some sort of addon model that is based on schnell. A few other distilled as well.

wardensc2•10/9/24, 5:31 PM

To be more specific I train 2 loras with about 30 images and about 5800 step, result is no improvement

VigilenceOP•10/9/24, 5:32 PM

Same config?

dsienra•10/9/24, 5:32 PM

flux-dev-de-distill is the best for training I'm very impressed, then you can use the lora on regular Flux-dev any quantization as normal

VVigilence Same config?

wardensc2•10/9/24, 5:32 PM

yes

wardensc2•10/9/24, 5:32 PM

just change the checkpoint only

VigilenceOP•10/9/24, 5:32 PM

Maybe it needs different config since the data is distilled.

dsienra•10/9/24, 5:33 PM

the best option will be to full finetune flux-dev-de-distill and then extract the lora

VigilenceOP•10/9/24, 5:33 PM

I want to do that, but I doubt my 4090 can do it in a reasonable time.

VVigilence Maybe it needs different config since the data is distilled.

wardensc2•10/9/24, 5:34 PM

do you read the author said

wardensc2•10/9/24, 5:34 PM

he said he finetune with 3 million high quality data

wardensc2•10/9/24, 5:34 PM

I bet he need to have a company like facebook or amazon

wardensc2•10/9/24, 5:34 PM

to do that

wardensc2•10/9/24, 5:34 PM

= )))

wardensc2•10/9/24, 5:35 PM

I dont buy it at all

VVigilence Maybe it needs different config since the data is distilled.

dsienra•10/9/24, 5:35 PM

the config is the same for training, is different for inferring it uses real cfg rather than Distilled CFG Scale so is slower to generate pictures

wardensc2•10/9/24, 5:35 PM

base on how long to train just 30 images for a checkpoint

dsienra•10/9/24, 5:36 PM

for that reason I use the trained lora on regular flux-dev

VigilenceOP•10/9/24, 5:36 PM

Yeah I would like to extract the lora.

Wwardensc2 he said he finetune with 3 million high quality data

dsienra•10/9/24, 5:37 PM

that model is not de-distilled try this one https://huggingface.co/nyanko7/flux-dev-de-distill

nyanko7/flux-dev-de-distill · Hugging Face

VVigilence Yeah I would like to extract the lora.

wardensc2•10/9/24, 5:37 PM

training checkpoint of flux is require much better hardware than lora, so training a checkpoint then extract a lora is not an efficient way to get a lora. I rather train a lora directly

VigilenceOP•10/9/24, 5:37 PM

@wardensc2 I tried the 4090 config the Dr. had an it said about 4000 hours for 100 images.

VigilenceOP•10/9/24, 5:38 PM

96gb ram

VVigilence @wardensc2 I tried the 4090 config the Dr. had an it said about 4000 hours for 1...

wardensc2•10/9/24, 5:38 PM

it's true

VigilenceOP•10/9/24, 5:38 PM

I had to edit his lora config as well, was too slow.

wardensc2•10/9/24, 5:38 PM

I train an rank 128 lora fp16, I need 4 a6000 with 48gb Vram and nearly 5 hours

wardensc2•10/9/24, 5:38 PM

but I get the best quality lora I ever have

VigilenceOP•10/9/24, 5:39 PM

I can only do 64 rank since I need to work as well.

VigilenceOP•10/9/24, 5:39 PM

2.61 s/it

VVigilence I can only do 64 rank since I need to work as well.

wardensc2•10/9/24, 5:39 PM

that why we use runpod or mass compute

wardensc2•10/9/24, 5:40 PM

because we can get something like H100 or A100 which speed is double or quadruple of the 4090

VigilenceOP•10/9/24, 5:40 PM

Yeah I plan on switching but wanted to learn how to do everything before being charged $$$ for failures.

VVigilence @wardensc2 I tried the 4090 config the Dr. had an it said about 4000 hours for 1...

daroth•10/9/24, 5:40 PM

On my 4090 it took 2.5 hours to train a lora

wardensc2•10/9/24, 5:41 PM

when I use 4 a6000 basically I have a vga with 4x speed of a 3090 and 48gb Vram

VigilenceOP•10/9/24, 5:41 PM

@daroth How many epochs?

wardensc2•10/9/24, 5:41 PM

cost me only 4 usd for a lora

VigilenceOP•10/9/24, 5:41 PM

I train slow and go for around 20.

@hamuz I am using this one and enjoy the results. https://medium.com/@zhiwangshi28/why-flux-lora-so

Similar Threads

Similar Threads

Similar Threads