Has anyone ran any tests on different optimizers using Kohya ss? to see whatg provides best results?
Has anyone ran any tests on different optimizers using Kohya ss? to see whatg provides best results?
Dbeen testing for a week now, trained about 40 sdxl models, i'd say either go for adafactor or adamw8bit, I know some people have used Prodigy.
Here Test One with adamw8bit is white shirt sample
and test two adafactor is yellow shirt sample
Here Test One with adamw8bit is white shirt sample
and test two adafactor is yellow shirt sample
GGot it. It's hard to say which one is better off of this. Was thinking about trying Lion Optimizer because I liked that when training in A1111
Fbest way is upload into hugging face
Fthen download from there :d
Fye extension broken
Fyou need a specific version
F
Fon colab i never optimized parameters
Fit doesnt matter i think
Fi thin adafactor is best
Fbut it depends other config too
DWill also test it as well soon, these ones i trained straight on lora though, not on dreambooth
Flion was best with that extension correct
Fbut didnt work well for me with kohya
FDi've noticed more flexibility when uses multiple batch sizes like 3-4-5
Dthough some quality tradeoff
GYou provide in this best settings based on different GB sizes. Did you use runpod to do these tests?
GAlso is this for Loras or Dreambooth? I noticed the # of epochs is 4, that seems small.
Fyes i use them on runpod too
GGot it. Thank you for sharing
F4 epoch is for 40 repeat
Fso 160 epochs
F
PI'm training LoRa SDXL on my 3060 with 12GB VRAM, 768*768 resolution and it's showing 2 hours and 30 minutes of training time. I've used 15 images with 20 repeats. Is this configuration fine?
--num_cpu_threads_per_process=2 "./sdxl_train_network.py"
--network_alpha="1"
--save_model_as=safetensors
--network_module=networks.lora
--text_encoder_lr=0.0004
--unet_lr=0.0004
--network_dim=32
--output_name="local_test_1"
--lr_scheduler_num_cycles="8"
--no_half_vae
--learning_rate="0.0004"
--lr_scheduler="constant"
--train_batch_size="1"
--max_train_steps="4800"
--save_every_n_epochs="1"
--mixed_precision="fp16"
--save_precision="fp16"
--caption_extension=".txt"
--cache_latents
--cache_latents_to_disk
--optimizer_type="Adafactor"
--optimizer_args scale_parameter=False relative_step=False warmup_init=False
--max_data_loader_n_workers="0"
--bucket_reso_steps=64
--gradient_checkpointing
--full_fp16
--xformers
--bucket_no_upscale
--noise_offset=0.0
--lowram
--num_cpu_threads_per_process=2 "./sdxl_train_network.py"
--network_alpha="1"
--save_model_as=safetensors
--network_module=networks.lora
--text_encoder_lr=0.0004
--unet_lr=0.0004
--network_dim=32
--output_name="local_test_1"
--lr_scheduler_num_cycles="8"
--no_half_vae
--learning_rate="0.0004"
--lr_scheduler="constant"
--train_batch_size="1"
--max_train_steps="4800"
--save_every_n_epochs="1"
--mixed_precision="fp16"
--save_precision="fp16"
--caption_extension=".txt"
--cache_latents
--cache_latents_to_disk
--optimizer_type="Adafactor"
--optimizer_args scale_parameter=False relative_step=False warmup_init=False
--max_data_loader_n_workers="0"
--bucket_reso_steps=64
--gradient_checkpointing
--full_fp16
--xformers
--bucket_no_upscale
--noise_offset=0.0
--lowram
MIt's interesting, because I also have an RTX 3060 12 GB card and I can train a person Lora in 20 minutes on SD 1.5 with 512x512
MI would use 7 repeat and classification images with 10 epochs, no captions, 128/64 weights. For Adafactor use classic parameters for LR and Unet LR: 0.0001, TE: 5e-05, token length: 225, model: RealisticVision 5.1, optimizer args: scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01
PHow were the results obtained?
MI have also looked at the parameters from Furkan and other loras. I experimented with weights and trained on RealisticVision with Adafactor.
MHere is a few sample from yesterday model, and no used hires fix for generation now.
MLook her hands, it's not fixed, just pure prompt
JIs 6700 steps enough for sdxl?
Mand here is a few portrait
Mgenerates stable images even under aux10 model
MWith artUniverse model:
MAgree I made some test for speed/memory. No gradient checkpointing:
xformers + Mem Efficient Attention = xformers without Mem. E. attention
Same speed, same memory usage.
xformers + Mem Efficient Attention = xformers without Mem. E. attention
Same speed, same memory usage.
MI noticed a very interesting thing. I'm training for style and I haven't changed anything else, just the number of epochs from 10 to 30, and I get completely different sample images and the training values are absolutely different. I'm using Prodigy now.
Fyep looking decent
F512x512 much lower resolution
Fit is decent steps yes
Fyep i noticed same
Pbut result was not good 
Fi never got good results with Prodigy
Ftry bf16 too
Fbut reason could be 768x768
Falso add this parameter to optimizer : --max_grad_norm=0.0
Fi will do another research for LoRA
