and that was even on 64GB ram
and that was even on 64GB ram
Zso I know that was not the issue, lol
TSo for you it just started working? My was working, then i upgraded and can't get it working again
Zright, for me, no update in between. just a day or two, different instance, no changes
Zin my behavior, except checking that save state
Z(at least I'd like to think that was it)
TI have been using it on my other machine with 3090 but will for sure keep it off while i try to get this resolved on my older GPU
Zhaha, yeah, I moved training to Massed Compute
Zbecause couldn't stand the thought of blocking my GPU
Zfollowed once comment, do 140 epochs, then take that and do 60 more
ZI tricked my ex-wife
Z(I trained model on one of my kids)
Tyeah i had a 3090 for gaming, and now that I'm into AI, its cut into my gaming hobby.
Zyeah, used to be 80-150h/week in TFD or D2, now, nothing for the past two weeks
Zhaha
Zmaybe every two weeks
Z(these numbers seem high, lol)
Z
Tthats why I'm trying to get my 2nd GPU running.. just let that run all the time and free up primary PC.. plus i want to actually generate images and use all the cool AI stuff, but you also can't do that whie training.
Zessentially, fully train on 1 GPU on MC is 24-28h
Zso $10 a pop
Zworth for me
Zinstead of 5 days on home gpu
ZBut I get it
Tits taken me 6 versions to get my own likeness perfect.
LI have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_IT.json to train LORA? I had posted 8.94s with Doc's V9, but unfortunately after some times the training process failed due to lack of VRAM. Now I'm testing with Kohya original version and it reached 10.5 s/It. But it is more stable in VRAM usage.
C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:
with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]
C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:
torch.cpu.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cpu', args...) instead.with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]
ZYou should try the suggested approach. 140 Epochs (1 iteration per image) on dev model
Zthen take the 140th epoch
Zand take that as input model
Zand train 60 more epochs with saving every 10th
Zit is incredible
TZI was
Tyeah i generally find in a 200 epoch run, the best is somewhere between 120-160, but never the 200's or less than 100
ZDreambooth vs Lora, no question 
ZRight, when you take that sweet spot at 140, and use that as your base model for training for an additional 60 epochs
Zwith same dataset
T@Zet do you happen to know what it means to be overtrained/overcooked. I see this term all the time in reference to too many epoc/repeats, but what does it actually mean? is the 200th epoch usually overtrained?
Zovertrained means that you see it converge towards likeness, then go away from it
Zlike the model is trying too hard
Zand introduces hallucinations of you, i.e. it starts not looking like you at all
ZI only have family members trained, so cannot share
Zbut you will see it
Z(and only you can)
Zhaha
ZIt's like you crossed the canny valley, you have that sweet spot of confusion, and you right back in it
Zthat sweet spot is where you want to be
Ztoo far, you overtrained
Ti see, so it starts to head more like the early epochs. What i noticed, is with dreambooth, it starts with some "person" in memory that is simiar to some of your training photos, then it fine tunes that person into you epoch over epoch until its pretty much you at around 120
ZRight
LTo train Dreambooth, 24GB_GPU_23150MB_10.2_second_it_Tier_1.json with Doc V9 achieves 9.47s/it. It runs stable and smoothly.