Software Engineering Courses (SECourses)•10mo ago

Hi <@205854764540362752> , I need some advice as I have been struggling with Lora training for a pr

Hi @Furkan Gözükara SECourses ,

I need some advice as I have been struggling with Lora training for a product. I tried with and without captions (without was worse).

Problems I'm Facing

Trigger words don't work consistently. For example, if I use "KOKUMKF2512NVYP2," sometimes I get some kind of device (see trigger_word_image) and sometimes a dress but with different colors and the same pattern.
When I use prompts where I explain the dress and background, I get a blue dress, but each time the dress color appears in different shades.
Hands are distorted a lot.
When I prompt it to generate on different body types, it doesn't work.

## What I Have Tried So Far

No Caption Lora:

batch_size: 2
steps: 4000

batch_size: 2
steps: 4000

batch_size: 2
steps: 4000

batch_size: 2
steps: 4000

Captioned Lora (see screenshot for dataset):

batch_size: 2
steps: 3500
sample caption: a person seated on a wooden chair against a dark gray paneled wall. The background is simple with wooden flooring visible. The model is wearing KOKUMKF2512NVYP2 with a KOKUMKF2512NVYP2DUPATTA draped across their lap and extending to the floor. The model is accessorized with pearl-like earrings, bracelets, and embellished footwear. Their hands are positioned casually, with one resting on their lap and the other on the arm of the wooden chair. The setting appears to be an indoor studio environment with minimalist decor, highlighting the seated figure against the contrasting background.

batch_size: 2
steps: 3500
sample caption: a person seated on a wooden chair against a dark gray paneled wall. The background is simple with wooden flooring visible. The model is wearing KOKUMKF2512NVYP2 with a KOKUMKF2512NVYP2DUPATTA draped across their lap and extending to the floor. The model is accessorized with pearl-like earrings, bracelets, and embellished footwear. Their hands are positioned casually, with one resting on their lap and the other on the arm of the wooden chair. The setting appears to be an indoor studio environment with minimalist decor, highlighting the seated figure against the contrasting background.

batch_size: 2
steps: 3500
sample caption: a person seated on a wooden chair against a dark gray paneled wall. The background is simple with wooden flooring visible. The model is wearing KOKUMKF2512NVYP2 with a KOKUMKF2512NVYP2DUPATTA draped across their lap and extending to the floor. The model is accessorized with pearl-like earrings, bracelets, and embellished footwear. Their hands are positioned casually, with one resting on their lap and the other on the arm of the wooden chair. The setting appears to be an indoor studio environment with minimalist decor, highlighting the seated figure against the contrasting background.

batch_size: 2
steps: 3500
sample caption: a person seated on a wooden chair against a dark gray paneled wall. The background is simple with wooden flooring visible. The model is wearing KOKUMKF2512NVYP2 with a KOKUMKF2512NVYP2DUPATTA draped across their lap and extending to the floor. The model is accessorized with pearl-like earrings, bracelets, and embellished footwear. Their hands are positioned casually, with one resting on their lap and the other on the arm of the wooden chair. The setting appears to be an indoor studio environment with minimalist decor, highlighting the seated figure against the contrasting background.

Mixed Lora of cloth and model head (see screenshot for dataset):

batch_size: 1
steps: 4000
sample caption for model headshot image: The image shows a zoya_aurage looking off to the side with a neutral expression. The background is a solid gray color creating a minimal, studio-like setting. zoya_aurage is positioned in a profile-like pose with bare shoulders visible, creating a clean and professional portrait composition.

batch_size: 1
steps: 4000
sample caption for model headshot image: The image shows a zoya_aurage looking off to the side with a neutral expression. The background is a solid gray color creating a minimal, studio-like setting. zoya_aurage is positioned in a profile-like pose with bare shoulders visible, creating a clean and professional portrait composition.

batch_size: 1
steps: 4000
sample caption for model headshot image: The image shows a zoya_aurage looking off to the side with a neutral expression. The background is a solid gray color creating a minimal, studio-like setting. zoya_aurage is positioned in a profile-like pose with bare shoulders visible, creating a clean and professional portrait composition.

batch_size: 1
steps: 4000
sample caption for model headshot image: The image shows a zoya_aurage looking off to the side with a neutral expression. The background is a solid gray color creating a minimal, studio-like setting. zoya_aurage is positioned in a profile-like pose with bare shoulders visible, creating a clean and professional portrait composition.

trigger_word_image92.12KB

trigger_word_image_256.77KB

prompt_sample663B

umrfarooqOP•3/21/25, 9:06 PM

Thank you so much for doing amazing research and sharing with us. And Sorry for the little long message I just want to share as much detail as possible. And is full finetuning is a good choice for this specific use case or not?

Here is some more details:

For the dataset:
Dress: 16 images
Model/Person: 9 headshot images generated by comfy consistent character workflow

I also tried using both lora (dress and person) to create images in SwarmUI but results are disappointing. The major importance for me is to get the pattern of dress, color of dress and person wearing wearing that dress. Please let me know If you need any other details. Please guide me to the right direction. Thanks again

hau•3/21/25, 9:09 PM

@Furkan Gözükara SECourses hey, really enjoyed your videos.

would appreciate a deep-dive comparing the following methods:

full dreambooth on flux (extremely memory intensive)
dreambooth lora on flux
full dreambooth on flux + then extract the lora via SVD
plain flux lora
plan flux full finetune

Joshua A•3/22/25, 1:19 AM

@Furkan Gözükara SECourses I am trying out dreambooth training on runpod using your guide. I got it to run but out of memory on a 4090.

Any guidance would be helpful.

------------------------------------
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 MiB. GPU 0 has a total capacity of 23.64 GiB of which 20.81 MiB is free. Process 3037847 has 23.62 GiB memory in use. Of the allocated memory 23.08 GiB is allocated by PyTorch, and 86.61 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)"

Uumrfarooq Hi <@205854764540362752> , I need some advice as I have been struggling with Lo...

Furkan Gözükara SECourses•3/22/25, 2:07 AM

hey your trigger word is wrong. it is many multiple words combination

Furkan Gözükara SECourses•3/22/25, 2:07 AM

i recommend watch this video no skip

Furkan Gözükara SECourses•3/22/25, 2:07 AM

https://youtu.be/FvpWy1x5etM

YouTubeSECourses

FLUX Full Fine-Tuning / DreamBooth Training Master Tutorial for Win...

If you want to train FLUX with maximum possible quality, this is the tutorial looking for. In this comprehensive tutorial, you will learn how to install Kohya GUI and use it to fully Fine-Tune / DreamBooth FLUX model. After that how to use SwarmUI to compare generated checkpoints / models and find the very best one to generate most amazing image...

Hhau <@205854764540362752> hey, really enjoyed your videos. would appreciate a deep-...

Furkan Gözükara SECourses•3/22/25, 2:08 AM

i already did and full dreambooth on flux works as low as 6 GB GPUs

Furkan Gözükara SECourses•3/22/25, 2:08 AM

with 64 GB RAM : https://youtu.be/FvpWy1x5etM

YouTubeSECourses

FLUX Full Fine-Tuning / DreamBooth Training Master Tutorial for Win...

JJoshua A <@205854764540362752> I am trying out dreambooth training on runpod using your g...

Furkan Gözükara SECourses•3/22/25, 2:08 AM

yes which config you used?

Furkan Gözükara SECourses•3/22/25, 2:08 AM

and are your images all 1024x1024?

FFurkan Gözükara SECourses yes which config you used?

Joshua A•3/22/25, 2:14 AM

Yes. All images are 1024. The config is 24GB_GPU_22900MB_7.3_second_it_Tier_1.

Mentioning this because I think it's strange. I am noticing when I add the preset to kohya preset nothing changes. I have to go through and manually change everything. and when I save the config. It doesn't save. refreshing the browser will erase all data.

I have 200gb storage, 30 gb of local storage like you say in the video.

JJoshua A Yes. All images are 1024. The config is 24GB_GPU_22900MB_7.3_second_it_Tier_1. ...

Furkan Gözükara SECourses•3/22/25, 2:14 AM

that means you didnt load config accurately

Furkan Gözükara SECourses•3/22/25, 2:14 AM

it has to load everything

Furkan Gözükara SECourses•3/22/25, 2:15 AM

tell me your config path you tried to load

FFurkan Gözükara SECourses that means you didnt load config accurately

Joshua A•3/22/25, 2:17 AM

Let me check. this is also what I am running it on. pytorch:2.2.0-py3.10-cuda12.1.

Joshua A•3/22/25, 2:17 AM

Should I make sure the folders don't have spaces in the names? I keep remembering runpod doesn't like spaces.

JJoshua A Should I make sure the folders don't have spaces in the names? I keep rememberin...

Furkan Gözükara SECourses•3/22/25, 2:19 AM

yes never use space anywhere

Furkan Gözükara SECourses•3/22/25, 2:19 AM

runpod starts with

Furkan Gözükara SECourses•3/22/25, 2:19 AM

/workspace/ etc

Joshua A•3/22/25, 2:21 AM

/workspace/dreambooth/DreamBooth_Tab_Fine_Tuning_Best_FLUX_Configs/24GB_GPU_22900MB_7.3_second_it_Tier_1.json

FFurkan Gözükara SECourses yes never use space anywhere

Joshua A•3/22/25, 2:21 AM

It looks like the config file path is ok.

JJoshua A It looks like the config file path is ok.

Furkan Gözükara SECourses•3/22/25, 2:24 AM

yes you loaded into dreambooth tab

Furkan Gözükara SECourses•3/22/25, 2:24 AM

or lora tab

FFurkan Gözükara SECourses yes never use space anywhere

Joshua A•3/22/25, 2:24 AM

Can I use flux1-dev-fp8.safetensors for training or do I need to use the original?

JJoshua A Can I use flux1-dev-fp8.safetensors for training or do I need to use the origina...

Furkan Gözükara SECourses•3/22/25, 2:24 AM

use original

Joshua A•3/22/25, 2:24 AM

It was the dreambooth tab. I didn't use the original so I can try that out now.

JJoshua A It was the dreambooth tab. I didn't use the original so I can try that out now.

Furkan Gözükara SECourses•3/22/25, 2:25 AM

ye try it

FFurkan Gözükara SECourses ye try it

Joshua A•3/22/25, 2:29 AM

That one strange thing is the config file isn't doing anyhting. very strange to me. It didn't save any of the work from before.

Screenshot_2025-03-21_at_10.29.25_PM.png

JJoshua A That one strange thing is the config file isn't doing anyhting. very strange to ...

Furkan Gözükara SECourses•3/22/25, 2:29 AM

click both

Furkan Gözükara SECourses•3/22/25, 2:29 AM

Furkan Gözükara SECourses•3/22/25, 2:29 AM

it has to load values

Furkan Gözükara SECourses•3/22/25, 2:30 AM

FFurkan Gözükara SECourses click both

Joshua A•3/22/25, 2:38 AM

It seemed to have saved this time. What do I put if I only want to save the 160th epoch in the "Save every N epochs" .

Screenshot_2025-03-21_at_10.36.02_PM.png

Joshua A•3/22/25, 2:39 AM

this is what the config file has loaded minus the save every n epoch number. I added 160.

FFurkan Gözükara SECourses i already did and full dreambooth on flux works as low as 6 GB GPUs

hau•3/22/25, 2:51 AM

amazing --- where are you doing full dreambooth? which script in kohya does that.... not seeing it in the video

hau•3/22/25, 2:55 AM

there doesn't seem to be any script in kohya that lets you do dreambooth with flux

FFurkan Gözükara SECourses ye try it

Joshua A•3/22/25, 2:59 AM

I went with the 16gb config file and its working. it looks like there was something that was causing it to fail at 24gb.

I don't know if this matters but I am getting this during the training. however this was also happening during the 16gb training too.

2025-03-22 02:58:21.761881: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.

Hhau there doesn't seem to be any script in kohya that lets you do dreambooth with fl...

hau•3/22/25, 3:01 AM

this script will let you do full flux finetuning on low vram GPUs

https://github.com/kohya-ss/sd-scripts/blob/sd3/flux_train.py

but there is no script that does dreambooth training? so it seems like the video you posted is not using dreambooth -- please correct me if i am wrong

@Furkan Gözükara SECourses

GitHub

sd-scripts/flux_train.py at sd3 · kohya-ss/sd-scripts

Contribute to kohya-ss/sd-scripts development by creating an account on GitHub.

hau•3/22/25, 3:03 AM

unless this line implies using dreambooth: https://github.com/kohya-ss/sd-scripts/blob/sd3/flux_train.py#L88

GitHub

sd-scripts/flux_train.py at sd3 · kohya-ss/sd-scripts

Contribute to kohya-ss/sd-scripts development by creating an account on GitHub.

hau•3/22/25, 3:03 AM

wardensc2•3/22/25, 3:26 AM

Hi @Furkan Gözükara SECourses I just download your config for SDXL training. For training dreambooth for character how many regulation images from your database you think we need to use train to get good dreambooth character model

sai_150•3/22/25, 8:07 AM

Hi @Furkan Gözükara SECourses , I've 33 images of me. can you suggest how many repeats I should consider(config has 40 repeats and 1 epoch, that gives 1320 steps).

sai_150•3/22/25, 8:11 AM

what model should I use for the best results sdxl base or any custom model from Civitai (like realvis). Is this the best method for generating images of a specific person?

JJoshua A It seemed to have saved this time. What do I put if I only want to save the 160t...

Furkan Gözükara SECourses•3/22/25, 12:02 PM

my config doesnt have save every 160 epooch