its taken me 6 versions to get my own likeness perfect.
its taken me 6 versions to get my own likeness perfect.
LI have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_IT.json to train LORA? I had posted 8.94s with Doc's V9, but unfortunately after some times the training process failed due to lack of VRAM. Now I'm testing with Kohya original version and it reached 10.5 s/It. But it is more stable in VRAM usage.
C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:
with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]
C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:
torch.cpu.amp.autocast(args...)torch.amp.autocast('cpu', args...)with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]
ZYou should try the suggested approach. 140 Epochs (1 iteration per image) on dev model
Zthen take the 140th epoch
Zand take that as input model
Zand train 60 more epochs with saving every 10th
Zit is incredible
TZI was
Tyeah i generally find in a 200 epoch run, the best is somewhere between 120-160, but never the 200's or less than 100
ZDreambooth vs Lora, no question 
ZRight, when you take that sweet spot at 140, and use that as your base model for training for an additional 60 epochs
Zwith same dataset
T@Zet do you happen to know what it means to be overtrained/overcooked. I see this term all the time in reference to too many epoc/repeats, but what does it actually mean? is the 200th epoch usually overtrained?
Zovertrained means that you see it converge towards likeness, then go away from it
Zlike the model is trying too hard
Zand introduces hallucinations of you, i.e. it starts not looking like you at all
ZI only have family members trained, so cannot share
Zbut you will see it
Z(and only you can)
Zhaha
ZIt's like you crossed the canny valley, you have that sweet spot of confusion, and you right back in it
Zthat sweet spot is where you want to be
Ztoo far, you overtrained
Ti see, so it starts to head more like the early epochs. What i noticed, is with dreambooth, it starts with some "person" in memory that is simiar to some of your training photos, then it fine tunes that person into you epoch over epoch until its pretty much you at around 120
ZRight
LTo train Dreambooth, 24GB_GPU_23150MB_10.2_second_it_Tier_1.json with Doc V9 achieves 9.47s/it. It runs stable and smoothly.
Zwell, it depends on the quality of the dataset
Zfor me, it was fun with one of my sons
Zbecause I included several ages without desciption
Tthats impressive. i've only gotten 12.18/it
Zso, sometimes, its a younger version, especially in earlier epochs
Lwith V9 version?
Zbut right now, with the 2 tiered training, I succesfully fooled his mom (inadvertently)
ZShe asked if it was his school picture, because she would love if he could wear that suit to a weeding next weekend
Zlol
Ti've had ones where for some reason it started with a really old guy with wrinkles and everything, than as it go closer to me, it slowly de-aged him, but still left remnants of the original old guy. Another time it chose another pic and started with a chubby guy, and left remanents of a bigger guy. Its all random, but i think you can get better results based on which person it starts with.
Tyes thats the test of a good lora.. if you can fool you mom, and i did.
Ti didn't upgrade the 3090, i think its only v8. been trying to get my 2080 working again.. i guess v9 is even faster?
LTry set “Blocks to swap" =9 or 8
LLkeep in mind that the more you lower this value the more VRAM it will consume. Play with values between 7 and 10
F23500MB is a lot . you really need to free up 3090 GPU for it
Fthis one is great and your speed is too
Wsame at my 3090 speed even at that time I dont upgrade to Torch 2.5
Wafter download your newest dreambooth training config, I will test whether speed improve or not
Where is my speed before update
W
Fconfig same only spda changed to xformers
LI noticed that the speed gain is in the best GPU utilization. GPU utilization is at 100% constantly. It's great, but be careful to keep your equipment well cooled. In my case the VRAM temp went up to 94 celsius after 7 hours of training.
