Software Engineering Courses (SECourses)•15mo ago

I have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_I

I have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_IT.json to train LORA? I had posted 8.94s with Doc's V9, but unfortunately after some times the training process failed due to lack of VRAM. Now I'm testing with Kohya original version and it reached 10.5 s/It. But it is more stable in VRAM usage.

C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:

torch.cpu.amp.autocast(args...)

torch.cpu.amp.autocast(args...)

torch.cpu.amp.autocast(args...)

torch.cpu.amp.autocast(args...) is deprecated. Please use

torch.amp.autocast('cpu', args...)

torch.amp.autocast('cpu', args...)

torch.amp.autocast('cpu', args...)

torch.amp.autocast('cpu', args...) instead.
with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]

TtheJindesan its taken me 6 versions to get my own likeness perfect.

Zet•10/26/24, 2:56 AM

You should try the suggested approach. 140 Epochs (1 iteration per image) on dev model

Zet•10/26/24, 2:56 AM

then take the 140th epoch

Zet•10/26/24, 2:56 AM

and take that as input model

Zet•10/26/24, 2:56 AM

and train 60 more epochs with saving every 10th

Zet•10/26/24, 2:56 AM

it is incredible

theJindesan•10/26/24, 2:56 AM

@Leolis78 ah, i thought you were doing the dreambooth method.. I do get faster lora training/it.

Zet•10/26/24, 2:57 AM

I was

ZZet then take the 140th epoch

theJindesan•10/26/24, 2:57 AM

yeah i generally find in a 200 epoch run, the best is somewhere between 120-160, but never the 200's or less than 100

Zet•10/26/24, 2:57 AM

Dreambooth vs Lora, no question

TtheJindesan yeah i generally find in a 200 epoch run, the best is somewhere between 120-160,...

Zet•10/26/24, 2:58 AM

Right, when you take that sweet spot at 140, and use that as your base model for training for an additional 60 epochs

Zet•10/26/24, 2:58 AM

with same dataset

theJindesan•10/26/24, 2:59 AM

@Zet do you happen to know what it means to be overtrained/overcooked. I see this term all the time in reference to too many epoc/repeats, but what does it actually mean? is the 200th epoch usually overtrained?

Zet•10/26/24, 3:00 AM

overtrained means that you see it converge towards likeness, then go away from it

Zet•10/26/24, 3:00 AM

like the model is trying too hard

Zet•10/26/24, 3:00 AM

and introduces hallucinations of you, i.e. it starts not looking like you at all

Zet•10/26/24, 3:01 AM

I only have family members trained, so cannot share

Zet•10/26/24, 3:01 AM

but you will see it

Zet•10/26/24, 3:01 AM

(and only you can)

Zet•10/26/24, 3:01 AM

haha

Zet•10/26/24, 3:02 AM

It's like you crossed the canny valley, you have that sweet spot of confusion, and you right back in it

Zet•10/26/24, 3:02 AM

that sweet spot is where you want to be

Zet•10/26/24, 3:02 AM

too far, you overtrained

ZZet and introduces hallucinations of you, i.e. it starts not looking like you at all

theJindesan•10/26/24, 3:02 AM

i see, so it starts to head more like the early epochs. What i noticed, is with dreambooth, it starts with some "person" in memory that is simiar to some of your training photos, then it fine tunes that person into you epoch over epoch until its pretty much you at around 120

Zet•10/26/24, 3:02 AM

Right

TtheJindesan <@1141070068344688660> ah, i thought you were doing the dreambooth method.. I do...

Leolis78OP•10/26/24, 3:03 AM

To train Dreambooth, 24GB_GPU_23150MB_10.2_second_it_Tier_1.json with Doc V9 achieves 9.47s/it. It runs stable and smoothly.

Zet•10/26/24, 3:03 AM

well, it depends on the quality of the dataset

Zet•10/26/24, 3:03 AM

for me, it was fun with one of my sons

Zet•10/26/24, 3:03 AM

because I included several ages without desciption

LLeolis78 To train Dreambooth, 24GB_GPU_23150MB_10.2_second_it_Tier_1.json with Doc V9 ach...

theJindesan•10/26/24, 3:03 AM

thats impressive. i've only gotten 12.18/it

Zet•10/26/24, 3:03 AM

so, sometimes, its a younger version, especially in earlier epochs

TtheJindesan thats impressive. i've only gotten 12.18/it

Leolis78OP•10/26/24, 3:03 AM

with V9 version?

Zet•10/26/24, 3:04 AM

but right now, with the 2 tiered training, I succesfully fooled his mom (inadvertently)

Zet•10/26/24, 3:04 AM

She asked if it was his school picture, because she would love if he could wear that suit to a weeding next weekend

Zet•10/26/24, 3:04 AM

lol

ZZet so, sometimes, its a younger version, especially in earlier epochs

theJindesan•10/26/24, 3:05 AM

i've had ones where for some reason it started with a really old guy with wrinkles and everything, than as it go closer to me, it slowly de-aged him, but still left remnants of the original old guy. Another time it chose another pic and started with a chubby guy, and left remanents of a bigger guy. Its all random, but i think you can get better results based on which person it starts with.

theJindesan•10/26/24, 3:05 AM

yes thats the test of a good lora.. if you can fool you mom, and i did.

LLeolis78 with V9 version?

theJindesan•10/26/24, 3:06 AM

i didn't upgrade the 3090, i think its only v8. been trying to get my 2080 working again.. i guess v9 is even faster?

TtheJindesan thats impressive. i've only gotten 12.18/it

Leolis78OP•10/26/24, 3:06 AM

Try set “Blocks to swap" =9 or 8

TtheJindesan i didn't upgrade the 3090, i think its only v8. been trying to get my 2080 work...

Leolis78OP•10/26/24, 3:10 AM

Try with last version of @Furkan Gözükara SECourses Kohya_FLUX_DreamBooth_v9.zip .

LLeolis78 Try set “Blocks to swap" =9 or 8

Leolis78OP•10/26/24, 3:10 AM

keep in mind that the more you lower this value the more VRAM it will consume. Play with values between 7 and 10

LLeolis78 I have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23...

Furkan Gözükara SECourses•10/26/24, 10:14 AM

23500MB is a lot . you really need to free up 3090 GPU for it

LLeolis78 To train Dreambooth, 24GB_GPU_23150MB_10.2_second_it_Tier_1.json with Doc V9 ach...

Furkan Gözükara SECourses•10/26/24, 10:14 AM

this one is great and your speed is too

FFurkan Gözükara SECourses this one is great and your speed is too

wardensc2•10/26/24, 11:36 AM

same at my 3090 speed even at that time I dont upgrade to Torch 2.5

Wwardensc2 same at my 3090 speed even at that time I dont upgrade to Torch 2.5

Furkan Gözükara SECourses•10/26/24, 11:47 AM

nice

wardensc2•10/26/24, 12:01 PM

after download your newest dreambooth training config, I will test whether speed improve or not

wardensc2•10/26/24, 12:01 PM

here is my speed before update

wardensc2•10/26/24, 12:01 PM

Wwardensc2 Click to see attachment

Furkan Gözükara SECourses•10/26/24, 12:14 PM

config same only spda changed to xformers

FFurkan Gözükara SECourses this one is great and your speed is too

Leolis78OP•10/26/24, 4:42 PM

I noticed that the speed gain is in the best GPU utilization. GPU utilization is at 100% constantly. It's great, but be careful to keep your equipment well cooled. In my case the VRAM temp went up to 94 celsius after 7 hours of training.

Deleted User•10/26/24, 4:46 PM

I get like 6s/it on a 4090 and my GPU stays around 60-70C

I have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_I

Similar Threads