Software Engineering Courses (SECourses)•15mo ago

this is the difference between the outputs of a lora made with flux gym (more pink one) vs the one m

this is the difference between the outputs of a lora made with flux gym (more pink one) vs the one made with your suggested settings in koyha -- do you have any suggestions for changes to settings that might help?

Dazzastrous [4090]•10/27/24, 9:59 PM

whats the spoiler thing

Dazzastrous [4090]•10/27/24, 9:59 PM

lol

SaxonOP•10/27/24, 10:01 PM

it just seems less obnoxious to make the picture viewing opt-in

Dazzastrous [4090]•10/27/24, 10:05 PM

white-background-X0UKDkmjSG-kQGWxmNy8Ig-WxRu69-vS5KZHYLmX7Qikg.png

SSaxon this is the difference between the outputs of a lora made with flux gym (more pi...

Furkan Gözükara SECourses•10/27/24, 10:19 PM

when training style dataset matters most

Furkan Gözükara SECourses•10/27/24, 10:19 PM

did you read my style training guide?

Furkan Gözükara SECourses•10/27/24, 10:21 PM

selected 189 images for the client

Furkan Gözükara SECourses•10/27/24, 10:21 PM

lets see

SaxonOP•10/27/24, 10:21 PM

I didn't go through it comprehensively, but I did look at it

SSaxon I didn't go through it comprehensively, but I did look at it

Furkan Gözükara SECourses•10/27/24, 10:24 PM

so dataset matters most

Furkan Gözükara SECourses•10/27/24, 10:24 PM

and consistency

Furkan Gözükara SECourses•10/27/24, 10:24 PM

if not consistent captions help

Vigilence•10/27/24, 11:00 PM

@Saxon How many image repeats did you use in kohya? Default is 40 in kohya, 10 in fluxgym. You can look into the flux gym folder and see the settings used for your lora.

SaxonOP•10/27/24, 11:00 PM

I used the default of 10 in flux gym, and 1 in Kohya

Vigilence•10/27/24, 11:03 PM

I think that is a major difference, not enough time to learn the style. I have noticed this myself. The major drawback about fine tuning for styles, I think, is that it needs more image repeats, but I haven’t done enough testing yet. Was the dataset 1:1?

SaxonOP•10/27/24, 11:06 PM

yes, the data set was the same. I'll give it another try with higher repeats in kohya and see if it's the difference.

SaxonOP•10/27/24, 11:06 PM

The default epochs in gym is 16 -- is that how many epochs I would put into Koyha to try and replicate results? It seems low

SSaxon The default epochs in gym is 16 -- is that how many epochs I would put into Koyh...

Furkan Gözükara SECourses•10/27/24, 11:08 PM

default number of epochs really meaningless

Furkan Gözükara SECourses•10/27/24, 11:08 PM

it depends on each dataset

Furkan Gözükara SECourses•10/27/24, 11:08 PM

how many images you have total?

SaxonOP•10/27/24, 11:10 PM

21, but the style is very consistent. I could add more but I'm just trying to get a feel for the settings

Vigilence•10/27/24, 11:10 PM

You really need to look at the training log in flux gym, all the settings are different. I believe it uses Adam8bit, and a slightly faster learning rate. I also think training is in fp8.

Vigilence•10/27/24, 11:11 PM

@Furkan Gözükara SECourses do you have comparison of training with fp16 vs fp8?

Vigilence•10/27/24, 11:12 PM

@Saxon All the settings in the flux gym log can be transferred to kohya (manually) as it’s backend is kohya.

SaxonOP•10/27/24, 11:14 PM

are you referring to the train.bat file?

Vigilence•10/27/24, 11:15 PM

I think that’s it. If you right click and open in a text program you can see the settings.

Vigilence•10/27/24, 11:15 PM

Did you assign a trigger word during training/captioning and use it in both images when testing?

SaxonOP•10/27/24, 11:17 PM

here it is if that helps. They do have a trigger word, and I usually caption them with only the trigger word and the animal in the image

train.bat1.53KB

SSaxon 21, but the style is very consistent. I could add more but I'm just trying to ge...

Furkan Gözükara SECourses•10/28/24, 12:03 AM

you should add more

VVigilence <@205854764540362752> do you have comparison of training with fp16 vs fp8?

Furkan Gözükara SECourses•10/28/24, 12:04 AM

we cant train fp8

Furkan Gözükara SECourses•10/28/24, 12:04 AM

all trainings are done at bf16

Furkan Gözükara SECourses•10/28/24, 12:04 AM

kohya doesnt know how to do fp8 training :d

wardensc2•10/28/24, 12:28 AM

new Kohya with Torch 2.5 speed increase about 20% - 25%

wardensc2•10/28/24, 12:28 AM

wardensc2•10/28/24, 12:28 AM

I used 3090

Mmasood anyone have got some good flux-dev-de-destill OneTrainer finetuning configuratio...

masood•10/28/24, 5:47 AM

Wwardensc2 new Kohya with Torch 2.5 speed increase about 20% - 25%

Furkan Gözükara SECourses•10/28/24, 10:19 AM

amazing

Mmasood ?

Furkan Gözükara SECourses•10/28/24, 10:20 AM

i dont have yet. our kohya configs are transferable to onetrainer

masood•10/28/24, 10:24 AM

where can i find that khoya configs

Mmasood where can i find that khoya configs

Furkan Gözükara SECourses•10/28/24, 10:28 AM

Full Instructions, Configs, Installers, Information and Links Shared Post (the one used in the tutorial)

https://www.patreon.com/posts/112099700

Patreon

Kohya FLUX Fine Tuning (Full Checkpoints) Training Full Tutorial Fo...

Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon

Furkan Gözükara SECourses•10/28/24, 10:28 AM

https://youtu.be/FvpWy1x5etM

YouTubeSECourses

FLUX Full Fine-Tuning / DreamBooth Training Master Tutorial for Win...

If you want to train FLUX with maximum possible quality, this is the tutorial looking for. In this comprehensive tutorial, you will learn how to install Kohya GUI and use it to fully Fine-Tune / DreamBooth FLUX model. After that how to use SwarmUI to compare generated checkpoints / models and find the very best one to generate most amazing image...

Furkan Gözükara SECourses•10/28/24, 10:28 AM

it has both lora and fine tuning configs

masood•10/28/24, 10:28 AM

thank you Dr.

Vigilence•10/28/24, 8:30 PM

@Furkan Gözükara SECourses Can Kohya do sd 3.5 yet?
Here is my latest research.

Vigilence•10/28/24, 8:31 PM

@Saxon The training parameters are all here. Just match them in kohya. The lora network dim was trained at rank 4 btw.

accelerate launch ^
--mixed_precision bf16 ^
--num_cpu_threads_per_process 1 ^
sd-scripts/flux_train_network.py ^
--pretrained_model_name_or_path "C:\pinokio\api\fluxgym.git\models\unet\flux1-dev.sft" ^
--clip_l "C:\pinokio\api\fluxgym.git\models\clip\clip_l.safetensors" ^
--t5xxl "C:\pinokio\api\fluxgym.git\models\clip\t5xxl_fp16.safetensors" ^
--ae "C:\pinokio\api\fluxgym.git\models\vae\ae.sft" ^
--cache_latents_to_disk ^
--save_model_as safetensors ^
--sdpa --persistent_data_loader_workers ^
--max_data_loader_n_workers 2 ^
--seed 42 ^
--gradient_checkpointing ^
--mixed_precision bf16 ^
--save_precision bf16 ^
--network_module networks.lora_flux ^
--network_dim 4 ^
--optimizer_type adafactor ^
--optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" ^
--split_mode ^
--network_args "train_blocks=single" ^
--lr_scheduler constant_with_warmup ^
--max_grad_norm 0.0 ^--sample_prompts="C:\pinokio\api\fluxgym.git\outputs\k44\sample_prompts.txt" --sample_every_n_steps="500" ^
--learning_rate 8e-4 ^
--cache_text_encoder_outputs ^
--cache_text_encoder_outputs_to_disk ^
--fp8_base ^
--highvram ^
--max_train_epochs 16 ^
--save_every_n_epochs 4 ^
--dataset_config "C:\pinokio\api\fluxgym.git\outputs\k44\dataset.toml" ^
--output_dir "C:\pinokio\api\fluxgym.git\outputs\k44" ^
--output_name k44 ^
--timestep_sampling shift ^
--discrete_flow_shift 3.1582 ^
--model_prediction_type raw ^
--guidance_scale 1 ^
--loss_type l2 ^

VVigilence <@205854764540362752> Can Kohya do sd 3.5 yet? Here is my latest research.

Furkan Gözükara SECourses•10/28/24, 8:39 PM

he is actively working on but i havent tested yet

VVigilence <@146819718571884544> The training parameters are all here. Just match them in ...

SaxonOP•10/28/24, 10:12 PM

I went through and changed all the settings I could find, but I wasn't sure how to change the network_dim or network_args. I'm also not clear on the epochs, so I just set it to 16 because I'm not sure what the distinction is between epochs and max_train_epochs.

Vigilence•10/28/24, 10:41 PM

I believe network dim id lora rank and alpha, which should be changed from 128 to what you see there. Epoch is like a turn, max epoch is max turns you want. The kohya_ss GitHub wiki tells a bit more about the settings.

Vigilence•10/28/24, 10:42 PM

@Furkan Gözükara SECourses is it possible to merge flux models together?

VVigilence <@205854764540362752> is it possible to merge flux models together?

Furkan Gözükara SECourses•10/28/24, 10:45 PM

loras yes

this is the difference between the outputs of a lora made with flux gym (more pink one) vs the one m

Similar Threads