Software Engineering Courses (SECourses)•3y ago

Guys. My RTX3060 12gb is very limited. Can’t really train SDXL models on it and SD1.5 dreambooth tra

Guys. My RTX3060 12gb is very limited. Can’t really train SDXL models on it and SD1.5 dreambooth training takes overnight. Shoul I try some cloud services? How much vram does Kaggle give? Is it enough for SDXL model training? What do you guys use?

FFurkan Gözükara SECourses there is no formula

rafraf•10/8/23, 11:53 PM

cool

rafraf•10/9/23, 12:27 AM

by the way, @Dr. Furkan Gözükara you said text encoder is by default turned off, on kohya_ss sdxl. So, does it matter if i set text encoder LR to 0?

rafraf•10/9/23, 12:27 AM

Shouldnt affect the trainging correct?

Gerdes of Earth•10/9/23, 12:39 AM

Has anyone ran any tests on different optimizers using Kohya ss? to see whatg provides best results?

GGerdes of Earth Has anyone ran any tests on different optimizers using Kohya ss? to see whatg pr...

Deleted User•10/9/23, 12:54 AM

been testing for a week now, trained about 40 sdxl models, i'd say either go for adafactor or adamw8bit, I know some people have used Prodigy.

Here Test One with adamw8bit is white shirt sample
and test two adafactor is yellow shirt sample

DDeleted User been testing for a week now, trained about 40 sdxl models, i'd say either go for...

Gerdes of Earth•10/9/23, 12:55 AM

Got it. It's hard to say which one is better off of this. Was thinking about trying Lion Optimizer because I liked that when training in A1111

EEyeSpyBekoAI Hey gang, does anyone have experience on downloading safetensor files from runpo...

Furkan Gözükara SECourses•10/9/23, 12:57 AM

best way is upload into hugging face

Furkan Gözükara SECourses•10/9/23, 12:57 AM

then download from there :d

Furkan Gözükara SECourses•10/9/23, 12:58 AM

https://twitter.com/GozukaraFurkan/status/1710285956111057011

Kkesenteboregi and now 😦

Furkan Gözükara SECourses•10/9/23, 12:58 AM

ye extension broken

Furkan Gözükara SECourses•10/9/23, 12:58 AM

you need a specific version

Furkan Gözükara SECourses•10/9/23, 12:58 AM

here my auto installer : https://www.patreon.com/posts/auto-installer-84773926

NNeuromasters Are there any parameters that are not up to date?

Furkan Gözükara SECourses•10/9/23, 12:59 AM

on colab i never optimized parameters

Rrafraf by the way, @Dr. Furkan Gözükara you said text encoder is by default turned off,...

Furkan Gözükara SECourses•10/9/23, 12:59 AM

it doesnt matter i think

GGerdes of Earth Has anyone ran any tests on different optimizers using Kohya ss? to see whatg pr...

Furkan Gözükara SECourses•10/9/23, 12:59 AM

yes

Furkan Gözükara SECourses•10/9/23, 12:59 AM

i thin adafactor is best

Furkan Gözükara SECourses•10/9/23, 12:59 AM

but it depends other config too

GGerdes of Earth Got it. It's hard to say which one is better off of this. Was thinking about try...

Deleted User•10/9/23, 1:00 AM

Will also test it as well soon, these ones i trained straight on lora though, not on dreambooth

GGerdes of Earth Got it. It's hard to say which one is better off of this. Was thinking about try...

Furkan Gözükara SECourses•10/9/23, 1:00 AM

lion was best with that extension correct

Furkan Gözükara SECourses•10/9/23, 1:00 AM

but didnt work well for me with kohya

Furkan Gözükara SECourses•10/9/23, 1:00 AM

i shared all my testings in this post : https://www.patreon.com/posts/89213064

Deleted User•10/9/23, 1:01 AM

i've noticed more flexibility when uses multiple batch sizes like 3-4-5

Deleted User•10/9/23, 1:01 AM

though some quality tradeoff

FFurkan Gözükara SECourses lion was best with that extension correct

Gerdes of Earth•10/9/23, 1:03 AM

You provide in this best settings based on different GB sizes. Did you use runpod to do these tests?

Gerdes of Earth•10/9/23, 1:06 AM

Also is this for Loras or Dreambooth? I noticed the # of epochs is 4, that seems small.

Furkan Gözükara SECourses•10/9/23, 1:06 AM

https://www.youtube.com/watch?v=EEV8RPohsbw

YouTubeSECourses

How To Do Stable Diffusion XL (SDXL) DreamBooth Training (Full Fine...

GGerdes of Earth You provide in this best settings based on different GB sizes. Did you use runpo...

Furkan Gözükara SECourses•10/9/23, 1:06 AM

yes i use them on runpod too

Gerdes of Earth•10/9/23, 1:07 AM

Got it. Thank you for sharing

GGerdes of Earth Also is this for Loras or Dreambooth? I noticed the # of epochs is 4, that seems...

Furkan Gözükara SECourses•10/9/23, 1:07 AM

4 epoch is for 40 repeat

Furkan Gözükara SECourses•10/9/23, 1:07 AM

so 160 epochs

Furkan Gözükara SECourses•10/9/23, 1:07 AM

here dreambooth quick tutorial for patreon supporters : https://www.youtube.com/watch?v=EEV8RPohsbw

YouTubeSECourses

How To Do Stable Diffusion XL (SDXL) DreamBooth Training (Full Fine...

Praveen Miriyala•10/9/23, 2:28 AM

I'm training LoRa SDXL on my 3060 with 12GB VRAM, 768*768 resolution and it's showing 2 hours and 30 minutes of training time. I've used 15 images with 20 repeats. Is this configuration fine?

--num_cpu_threads_per_process=2 "./sdxl_train_network.py"
--network_alpha="1"
--save_model_as=safetensors
--network_module=networks.lora
--text_encoder_lr=0.0004
--unet_lr=0.0004
--network_dim=32
--output_name="local_test_1"
--lr_scheduler_num_cycles="8"
--no_half_vae
--learning_rate="0.0004"
--lr_scheduler="constant"
--train_batch_size="1"
--max_train_steps="4800"
--save_every_n_epochs="1"
--mixed_precision="fp16"
--save_precision="fp16"
--caption_extension=".txt"
--cache_latents
--cache_latents_to_disk
--optimizer_type="Adafactor"
--optimizer_args scale_parameter=False relative_step=False warmup_init=False
--max_data_loader_n_workers="0"
--bucket_reso_steps=64
--gradient_checkpointing
--full_fp16
--xformers
--bucket_no_upscale
--noise_offset=0.0
--lowram

SSuperTurboHero Guys. My RTX3060 12gb is very limited. Can’t really train SDXL models on it and ...

mikemenders•10/9/23, 4:52 AM

It's interesting, because I also have an RTX 3060 12 GB card and I can train a person Lora in 20 minutes on SD 1.5 with 512x512

PPraveen Miriyala I'm training LoRa SDXL on my 3060 with 12GB VRAM, 768*768 resolution and it's sh...

mikemenders•10/9/23, 4:56 AM

I would use 7 repeat and classification images with 10 epochs, no captions, 128/64 weights. For Adafactor use classic parameters for LR and Unet LR: 0.0001, TE: 5e-05, token length: 225, model: RealisticVision 5.1, optimizer args: scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01

Praveen Miriyala•10/9/23, 5:00 AM

How were the results obtained?

mikemenders•10/9/23, 5:02 AM

I have also looked at the parameters from Furkan and other loras. I experimented with weights and trained on RealisticVision with Adafactor.

mikemenders•10/9/23, 5:08 AM

Here is a few sample from yesterday model, and no used hires fix for generation now.

40436-2240229857-bbl_woman_red_dress_sitting_in_a_chair_restaurant_lipstick_some_man_in_background_upper_body_shot_top_view_look_at_vie.png

40440-1223372462-bbl_woman_red_dress_sitting_in_a_chair_restaurant_lipstick_some_man_in_background_upper_body_shot_top_view_look_at_vie.png

40438-367875223-bbl_woman_red_dress_sitting_in_a_chair_restaurant_lipstick_some_man_in_background_upper_body_shot_top_view_look_at_vie.png

40437-4019934250-bbl_woman_red_dress_sitting_in_a_chair_restaurant_lipstick_some_man_in_background_upper_body_shot_top_view_look_at_vie.png

mikemenders•10/9/23, 5:09 AM

Look her hands, it's not fixed, just pure prompt

40444-2754387671-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_upper_body_shot_top_view_look_at_viewer_and.png

40443-3481190311-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_upper_body_shot_top_view_look_at_viewer_and.png

40442-4031526407-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_upper_body_shot_top_view_look_at_viewer_and.png

40441-2800161235-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_upper_body_shot_top_view_look_at_viewer_and.png

JM•10/9/23, 5:10 AM

Is 6700 steps enough for sdxl?

mikemenders•10/9/23, 5:14 AM

and here is a few portrait

40450-2492411505-bbl_woman_blue_dress_short_hair_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many.png

40449-1065977894-bbl_woman_blue_dress_short_hair_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many.png

40447-1578977074-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sit.png

40446-1230343363-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sit.png

40445-1387945743-bbl_woman_green_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sit.png

mikemenders•10/9/23, 5:19 AM

generates stable images even under aux10 model

40464-2163930282-bbl_woman_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sitting_i.png

40463-1992744207-bbl_woman_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sitting_i.png

40462-1257673000-bbl_woman_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sitting_i.png

40461-1801482298-bbl_woman_dress_short_hair_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_peop.png

40460-1897398850-bbl_woman_dress_short_hair_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_peop.png

40459-1473175472-bbl_woman_dress_short_hair_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_peop.png

40458-3196843830-bbl_woman_dress_short_hair_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_peop.png

mikemenders•10/9/23, 5:26 AM

With artUniverse model:

40489-3077305307-bbl_woman_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sitting_i.png

40488-3117019385-bbl_woman_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sitting_i.png

40487-2920139718-bbl_woman_dress_sitting_in_a_chair_restaurant_some_man_in_background_portrait_look_at_viewer_and_many_people_sitting_i.png

FFurkan Gözükara SECourses i think memory efficient attention doing nothing 😄

Metaphysix•10/9/23, 5:39 AM

Agree I made some test for speed/memory. No gradient checkpointing:
xformers + Mem Efficient Attention = xformers without Mem. E. attention
Same speed, same memory usage.

mikemenders•10/9/23, 6:37 AM

I noticed a very interesting thing. I'm training for style and I haven't changed anything else, just the number of epochs from 10 to 30, and I get completely different sample images and the training values are absolutely different. I'm using Prodigy now.

PPraveen Miriyala I'm training LoRa SDXL on my 3060 with 12GB VRAM, 768*768 resolution and it's sh...

Furkan Gözükara SECourses•10/9/23, 9:27 AM

yep looking decent

Mmikemenders It's interesting, because I also have an RTX 3060 12 GB card and I can train a p...

Furkan Gözükara SECourses•10/9/23, 9:27 AM

512x512 much lower resolution