Software Engineering Courses (SECourses)•3y ago

Good morning, I have a question about training multiple characters in a single LORA. A few weeks ag

Good morning, I have a question about training multiple characters in a single LORA.

A few weeks ago, you mentioned that when captions are added to images (and these include the concept, for example, ohwx), the class token from the name of the directory where the images are stored is no longer used. That is, the 'ohwx' part of '20_ohwx' is not used, only the '20' is used to determine repetitions.

Based on this, if I want to train multiple characters, which strategy is better?
a) One folder for each character, even if they all have captions.
b) A single folder that encompasses all characters.

On another note, let's assume that for character1 I have 50 images and for character2 I have 100 images. Should this be taken into account in some way? For example, adjusting the repetitions so that the multiplication of repetition and number of images is similar across all datasets, or would this be counterproductive, and could it cause the dataset with fewer images and more repetitions to be overfitted compared to the rest?

Best regards and sorry for the long message

virginia•11/29/23, 10:05 AM

Furkan, here? /opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
usage: sdxl_train.py [-h] [--v2] [--v_parameterization]

virginia•11/29/23, 10:05 AM

always failed in checkpoint training

mikemenders•11/29/23, 12:31 PM

This is what a strong but flexible Lora graph looks like on Tensorboard. If a spike jumps out into the 3's that's borderline, but no more than 3's. So the goal is not to get max_norm/keys_scaled above 2. Plus a max_norm/max_key_norm should reach 1 before half of the training session, so you have time to normalize the model. So you can tell by looking at these values during training if the Lora will be good or not.

mikemenders•11/29/23, 12:32 PM

I trained this model with d_coef=0.85, because the first version (with 1.02) was strong, it went to 3 several times.

mikemenders•11/29/23, 12:37 PM

And the first tests showed that even this version was a bit strong.

EEvan has anyone tried this before/know how they are able to train so insanely fast?? ...

Furkan Gözükara SECourses•11/29/23, 1:11 PM

it is not faster

Furkan Gözükara SECourses•11/29/23, 1:11 PM

they use bigger batch size and lesser steps

Furkan Gözükara SECourses•11/29/23, 1:11 PM

actually when I tested before they did training on 512x512 haha

AAusent315 Good morning, I have a question about training multiple characters in a single L...

Furkan Gözükara SECourses•11/29/23, 1:12 PM

the second question is the answer. you want each dataset to be trained total number of steps equally

Furkan Gözükara SECourses•11/29/23, 1:12 PM

that is the logic of repeating

Furkan Gözükara SECourses•11/29/23, 1:12 PM

otherwise if that is a balanced dataset put all into a single folder

Vvirginia Furkan, here? /opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: Use...

Furkan Gözükara SECourses•11/29/23, 1:12 PM

this is not error

Furkan Gözükara SECourses•11/29/23, 1:12 PM

just a warning

Furkan Gözükara SECourses•11/29/23, 1:12 PM

make a video and send me

mikemenders•11/29/23, 1:14 PM

How much does the image set matter? 2 photos were duplicated, so my model may have been strong. After deleting these two and having only 23 photos left, I had to change the d_coef value, and 0.75 was too much. I went all the way down to 0.5, which seems to have been fine, although since it never reached the 1 limit of normalization, it was probably a bit under-trained. Based on the tests, not bad, but not reaching a score above 0.9, so really under-trained.

Mmikemenders How much does the image set matter? 2 photos were duplicated, so my model may ha...

Furkan Gözükara SECourses•11/29/23, 1:15 PM

i never got any useful info from these statistics for stable diffusion :/

virginia•11/29/23, 1:15 PM

okay

FFurkan Gözükara SECourses i never got any useful info from these statistics for stable diffusion :/

mikemenders•11/29/23, 1:19 PM

These graphs are only displayed when normalization is enabled for Lora (it is not enabled for DB). Based on the developer's description, I watch these two values to see if the maximum normalization reaches 1 (it is supposed to be good then) or not over normalizing, so how much it deviates from the original model, how much Lora needs to be improved. And from my observations, for cosine type training, these two graphs give very useful results under Prodigy.

Mmikemenders These graphs are only displayed when normalization is enabled for Lora (it is no...

Furkan Gözükara SECourses•11/29/23, 1:20 PM

Well I would prefer full DreamBooth and extract LoRA

mikemenders•11/29/23, 1:21 PM

Here is a topic on kohya site from normalization:
https://github.com/kohya-ss/sd-scripts/pull/545

GitHub

Dropout and Max Norm Regularization for LoRA training by AI-Casanov...

This PR adds Dropout and Max Norm Regularization [Paper] to train_network.py
Dropout randomly removes some weights/neurons from calculation on both the forward and backward passes, effectively trai...

mikemenders•11/29/23, 1:24 PM

That's what I was looking for in DreamBooth training, because it helps a lot to make a good model, and since I use it with cosine under Prodigy, I get absolutely good results when I watch the graph. I can see at a glance how good or bad Lora will be (see above), and tests have confirmed this.

BadNoise•11/29/23, 1:30 PM

Hi everyone,
I'm having serious problem with dreambooth, I always get terrible results and for the good ones, the face is not the same per dataset. I'm training everything on epicPhotogasm.

Of course it's my fault, this is my setup (apart from the A100 80G):
20 pictures of the same person (always portrait shot, not fullbody) in 1024x1024
2000 train steps
100 class images
lr 1e-6 100 validation steps
(please let me know if you need more details on the settings, but everything else is set as default)

Do you suggest to use 100/200 pictures and increase both train steps and class images? Or if you have better suggestions I'm open to everything!

Thank you!

mikemenders•11/29/23, 1:43 PM

And here's the graph of the winning Lora, with d_coef=0.6 and 23 images. While I like to train it a bit stronger, I also let the Hires Fix work, which I used to boost the initial face to 0.96 for the very first generation. You can see in the image that I achieved normalization 1 on the right graph (yellow color), so it made a good model and only adjusted the normalization a few times.

BBadNoise Hi everyone, I'm having serious problem with dreambooth, I always get terrible r...

Furkan Gözükara SECourses•11/29/23, 1:56 PM

are you using my patreon config?

FFurkan Gözükara SECourses are you using my patreon config?

BadNoise•11/29/23, 2:01 PM

unfortunately no

I sent you a dm btw, I'd appreciate if you can reply, thank you!

BBadNoise unfortunately no 😅 I sent you a dm btw, I'd appreciate if you can reply, thank...

Furkan Gözükara SECourses•11/29/23, 2:02 PM

checking dms in a moment hopefully sorry for delay

thewingag•11/29/23, 3:01 PM

RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
size mismatch for down_blocks.0.attentions.0.proj_in.weight: copying a param with shape
torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1])
my model: stable-diffusion-2-1-768v
resolution = 768
can u help m? thanks

DR.Siva•11/29/23, 3:01 PM

how to make resoltion (1024x1024) stick after every restart ? for image creation

Tthewingag RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel: s...

Furkan Gözükara SECourses•11/29/23, 3:03 PM

i really dont use sd 2.1 it is pretty useless

DDR.Siva how to make resoltion (1024x1024) stick after every restart ? for image creation

Furkan Gözükara SECourses•11/29/23, 3:04 PM

edit : ui-config.json

Furkan Gözükara SECourses•11/29/23, 3:04 PM

i should show this

DR.Siva•11/29/23, 3:05 PM

@Dr. Furkan Gözükara you can do basic guide on automatic1111 UI

DR.Siva•11/29/23, 3:05 PM

would set base for many guides imo. Also under settings there is so much to tweak and learn

Furkan Gözükara SECourses•11/29/23, 3:06 PM

i agree

DR.Siva•11/29/23, 3:10 PM

corrected

FFurkan Gözükara SECourses i agree

thewingag•11/29/23, 3:10 PM

should i use sd1.5?

DR.Siva•11/29/23, 3:13 PM

ok under defaults also you do the same i guess

DR.Siva•11/29/23, 3:13 PM

Tthewingag should i use sd1.5?

Furkan Gözükara SECourses•11/29/23, 3:19 PM

i prefer sdxl

FFurkan Gözükara SECourses i prefer sdxl

thewingag•11/29/23, 3:22 PM

yes. i often train sdxl on PC. but in train lora on colab free

Digital [Starburst]•11/29/23, 3:49 PM

I'm interested in a tutorial on how to set this up: https://stability.ai/news/stability-ai-sdxl-turbo

Stability AI

Introducing SDXL Turbo: A Real-Time Text-to-Image Generation Model ...

SDXL Turbo is a new text-to-image mode based on a novel distillation technique called Adversarial Diffusion Distillation (ADD), enabling the model to create image outputs in a single step and generate real-time text-to-image outputs while maintaining high sampling fidelity.

Digital [Starburst]•11/29/23, 3:50 PM

I got to try it for all of five seconds before reaching a usage cap. It was working great.

Digital [Starburst]•11/29/23, 3:51 PM

The image updates as you type making designing what you want much easier. There's no waiting 5 to 30 minutes to see the results.

Digital [Starburst]•11/29/23, 3:52 PM

Here's the Hugging Face link: https://huggingface.co/stabilityai/sdxl-turbo

stabilityai/sdxl-turbo · Hugging Face

Digital [Starburst]•11/29/23, 3:52 PM

In the meantime I have been trying to learn to use: https://stable-diffusion-art.com/deforum/

Stable Diffusion ArtAndrew

How to make a video with Stable Diffusion (Deforum) - Stable Diffus...

Deforum is a tool to create animation videos with Stable Diffusion. All you need to provide the prompts and settings for how the camera moves.

Digital [Starburst]•11/29/23, 3:53 PM

It's an amazing tool.

Digital [Starburst]•11/29/23, 3:53 PM

But very complex.

MrJoozy•11/29/23, 4:02 PM

wondering whats an estimate fine tune training time i can expect for using Nvidia 2070 Super for training a face with 5200 512x512 regularisation images and 40 training images with 40 repeats. using RealisticVision5.1 is it normal for 1 epoch to take 7 hours according to my tqdm? im also already using AdamW8bit

AIGambino•11/29/23, 5:01 PM

what is the significance of having the json file in my models folder with my custom train?

virginia•11/29/23, 5:44 PM

Furkan, can kaggle webui be used to do anmiatediff

Good morning, I have a question about training multiple characters in a single LORA. A few weeks ag

Similar Threads