Software Engineering Courses (SECourses)•3y ago

I'm trying to get a baseline for SDXL Lora time frames. I see a bunch of different times about train

I'm trying to get a baseline for SDXL Lora time frames. I see a bunch of different times about train a lora in 30 minutes etc but my use case is different. Most of my SDXL Lora trainings are 12-18 hours. Now I know this has to do in part im using 768X1344. Adjusting Network Dimm seems to reduce and learning rate seems to have an impact on the time but my concern quality will be impacted if I adjust these. My Anime models are coming out great despite the time. My person models are Meh despite following the guides and using the regularization images but I believe I missed something. I'm on an RTX 4090. Any insight?

FFurkan Gözükara SECourses huge. did you do anything special?

FiveSAI•12/9/23, 7:29 PM

Yep im using pagedadamw8bit, similar results to adafactor, still testing, got about 50 more trainings to test, will try prodigy next

LLorelai AI I'm trying to get a baseline for SDXL Lora time frames. I see a bunch of differe...

FiveSAI•12/9/23, 7:30 PM

can you share your json file, also are you using the latest nvidia drivers with Sysmem fallback policy

FFiveSAI can you share your json file, also are you using the latest nvidia drivers with ...

Lorelai AIOP•12/9/23, 7:33 PM

That’s a good idea to check the drivers. I’ll grab the json when I have a moment

FiveSAI•12/9/23, 7:33 PM

i also have a 4090 my trainings are usually 35 mins to 2 hours depending

Lorelai AIOP•12/9/23, 7:34 PM

What settings are you using? I know that’s a large request. I’m all over the place even with the trainings and I’m still new.

FiveSAI•12/9/23, 7:35 PM

depends i use different settings, but usually adafactor and the extra arguments with the gradient checkpointing and sometimes xformers not always.

FFiveSAI depends i use different settings, but usually adafactor and the extra arguments ...

Lorelai AIOP•12/9/23, 7:36 PM

Do you think adafactor is better than AdamW (not the 8 bit)

FiveSAI•12/9/23, 7:38 PM

For me for faster training with lora i always use adamw8bit, dreambooth still testing but best is adafactor

FiveSAI•12/9/23, 7:39 PM

and for sdxl i use the noise ofset 0.0357

FiveSAI•12/9/23, 7:40 PM

also using a older version of kohya uses less vram i think 21.8.4

FiveSAI•12/9/23, 7:42 PM

also if you want me to train your data set on my system i can. so we canm compare training times

Lorelai AIOP•12/9/23, 7:44 PM

I’m just playing around at the moment I’m going to look at the items you mentioned. What about the basic settings like images, epochs, repeats? What I’m struggling with real people. I followed the doctors videos and did the first 2 and got just ok results.

FiveSAI•12/9/23, 7:48 PM

So far people, I use 15-30 Images, no harsh angles , usually front, 3/4 and side shots.

20 Repeats seems to be the golden amount.

Also i found using too much class images affects my training,

and usually 8 epochs, I generate samples as well, so I usually stop the training early.

FiveSAI•12/9/23, 7:48 PM

Also you may want to try using min snr gamma at the value of 10

Lorelai AIOP•12/9/23, 7:50 PM

How many class images roughly?

FiveSAI•12/9/23, 7:51 PM

I have my own set of 2000, but it only uses around 500

the more repeats you have the more class images it uses

Lorelai AIOP•12/9/23, 7:51 PM

And what SDXL base model do you use?

FiveSAI•12/9/23, 7:52 PM

Its a bit tricky so far, i use the base sdxl model 1.0, But usually loras dont work too well with other checkpoints.

So im using a custom merge of base xl juggernaught xl dreamshaper and realvis,

FiveSAI•12/9/23, 7:53 PM

but if you're only use base xl, then yeah base xl is the best

FiveSAI•12/9/23, 7:54 PM

will probably make my own tutorial soon, still testing for best settings

Lorelai AIOP•12/9/23, 7:54 PM

I’m setting this up that the moment thanks so much for the help. Learning rates?

FiveSAI•12/9/23, 7:54 PM

Are you training lora?

Lorelai AIOP•12/9/23, 7:54 PM

Yes

FiveSAI•12/9/23, 7:55 PM

If you're using adafactor

0.0001
0.0001
0.0001

for all 3

FiveSAI•12/9/23, 7:56 PM

Extra arguments

scale_parameter=False relative_step=False warmup_init=False

FiveSAI•12/9/23, 7:56 PM

you can also try one with weight decay

scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01

Lorelai AIOP•12/9/23, 7:56 PM

Ok I think I have almost everything. Question though. The Doctor proves 5200 class images. Do I use the whole folder or pull some out.

FiveSAI•12/9/23, 8:00 PM

hmmm i havent used his ones, cause some of the images there you don't want to use.

So i'd just pull out the best ones from there, take about 1000>

get nice clear shot images