Software Engineering Courses (SECourses)•2y ago

Hi All, One question, i want to train stable diffusion to create pictures with background of local t

Hi All, One question, i want to train stable diffusion to create pictures with background of local tourist places. Any suggestion on how to train, technique etc.

SSathish H Hi All, One question, i want to train stable diffusion to create pictures with b...

Furkan Gözükara SECourses•12/31/23, 6:44 PM

you need to collect lots of good images

Furkan Gözükara SECourses•12/31/23, 6:44 PM

then you can llava caption them and train

Furkan Gözükara SECourses•12/31/23, 6:45 PM

or with activation token + caption

Furkan Gözükara SECourses•12/31/23, 6:45 PM

you should experiment

SSathish H Hi All, One question, i want to train stable diffusion to create pictures with b...

Lorelai AI•12/31/23, 6:56 PM

I'm about to do my first one I can give you feedback on how it went

FFurkan Gözükara SECourses or with activation token + caption

Sathish HOP•12/31/23, 7:26 PM

Thank you

tom•12/31/23, 9:28 PM

Hmm. Tried a few times. I keep getting the following error : orch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 14.76 GiB total capacity; 12.34 GiB already allocated; 3.75 MiB free; 13.47 GiB reserved in total by PyTorch. I've gone down to train just 9 1024x1024 images with this same error. Is there a way to flush the gpu or something? cudnn doesn't register.

tom•12/31/23, 9:28 PM

I'm re trying with lower repeats. Fingers crossed

Kallamamran•12/31/23, 9:40 PM

Happy New Year y'all!

Ttom Hmm. Tried a few times. I keep getting the following error : orch.cuda.OutOfMemo...

Furkan Gözükara SECourses•12/31/23, 9:59 PM

where are you trying what are you trying?

FFurkan Gözükara SECourses where are you trying what are you trying?

tom•12/31/23, 10:23 PM

Following the newer video. Instead of the male cell, used the female cell. Added 100 input images. Failed. Then tried again with 9 images. Failed. Then restarted everything with 9 images and no Regularisation images. This too failed Cuda out of memory error. Tried it again and it worked in about 36 min. (Obviously, this is just a test and the result not usable) I'm wondering if Kaggle is having some issues. Had errors yesterday as well.

tom•12/31/23, 10:23 PM

I'm going to try again shortly with a proper training. Fingers crossed.

tom•12/31/23, 10:25 PM

Any plans for a Comfyui via Kaggle class?

Ttom Any plans for a Comfyui via Kaggle class?

Furkan Gözükara SECourses•12/31/23, 11:18 PM

i tried but it didnt work on kaggle

tom•12/31/23, 11:46 PM

Ok, I will make an attempt at some comfyui via kaggle code after this training is done and let you know.

mjak•12/31/23, 11:47 PM

How does everyone here handle the repeats/epochs when doing the SDXL Training with Textencoder training?

The formula that @Dr. Furkan Gözükara mentions is:

Max Train Steps = Train_imags * 2(if class img used) * repeating_count * number_of_epochs

In the medium article @Dr. Furkan Gözükara mentions that it's 4500 steps at 150 repeating but using the formula above with regularization images = 9600 steps (15x2x40x8)

My dataset is 20 images so I've done 35 repeats per image and at 8 Epochs = 11,200 Steps
(20 images x 2 x 35 x 8)

mjak•12/31/23, 11:49 PM

I'm sure this has been covered elsewhere as it's a common challenge that Furkan complained about with Kohya

tom•12/31/23, 11:49 PM

I've taken the max train steps and / by 5 or 10 if the dataset is too big (100 training images)

mjak•12/31/23, 11:49 PM

@tom so what just decreasing the number of repeats?

tom•12/31/23, 11:50 PM

lemme check. i'm running one right now ...

tom•12/31/23, 11:54 PM

Under Data set prep : Training images repeats 100 | Regularisation Images 1 || Under Parameters>Advanced>Save every 3960 Steps. Latent Cache load time was ~30min. Training is currently saying 4hrs.

tom•12/31/23, 11:55 PM

A few hours ago, howerver, nothing was working. Maybe some random issue at kaggle. Didn't change any settings. But, even 9 images without any regularisation images was failing. Seems ok now. *knocks wood

tom•12/31/23, 11:56 PM

Epochs I have set to 10

tom•12/31/23, 11:58 PM

You could lower epochs while keeping repeats per image at 35

mjak•1/1/24, 12:20 AM

Yeah I guess I'm more interested in the thinking behind adjusting repeats + epochs depending on the training hyper params by @Dr. Furkan Gözükara

AIGambino•1/1/24, 12:21 AM

Dr wasn't it # of images x 40 x 2? = the total steps trained?

mjak•1/1/24, 12:22 AM

afaik kohya speaks about repeats being used to do variable weighting on images (makes sense) + combining training images with regularization images (either generated or ground truth like the unsplash data) so basically let's say 20 images 40 repeats x 2 (regularization = true) = 1600 steps per epoch / I guess maybe the thinking behind epochs is to slightly overfit the data anywyas and then compare what checkpoint you get at each epoch

Mmjak afaik kohya speaks about repeats being used to do variable weighting on images (...

tom•1/1/24, 1:00 AM

Yes, that would make sense. Olivio did a training a long while ago that kinda-sorta-didn't work until he did a "checkpoint merger" to a model that was very close to what he was training. I'm guessing that's the same with the regularisation images: Make them something like this, but greater emphasis on the the training image data.

John D.•1/1/24, 1:17 AM

Quick question - is it even possible to use the kohya to train SDXL LoRA on an 8gb GPU? If so, I already enabled "Gradient checkpointing" but still get out of memory errors - what else do I need to do?

Mmjak How does everyone here handle the repeats/epochs when doing the SDXL Training wi...

Furkan Gözükara SECourses•1/1/24, 1:19 AM

it is like this

Furkan Gözükara SECourses•1/1/24, 1:19 AM

15 images

Furkan Gözükara SECourses•1/1/24, 1:19 AM

150 repeat

Furkan Gözükara SECourses•1/1/24, 1:19 AM

1 epoch

Furkan Gözükara SECourses•1/1/24, 1:19 AM

2 from reg images

Furkan Gözükara SECourses•1/1/24, 1:19 AM