Twill try.. thank you again. its strange if its only supposed to use 32.2GB. why does mine keep climbing.. what is it loading into memory more?
Fye that doesnt make sense
Ffull training without shared vram uses 28.200 MB : 48GB_GPU_28200MB_6.4_second_it_Tier_1
TIDK, it keeps going over 32.2GB then it gives out of memory, on SPDA. I'll keep trying other things.
TIts like its not following the 35 blocks swap
ZYou didn't enable save state for some reason, right?
TI have in some configs, will that use more memory just to save a state? But i was also testing clean configs from the v9 zip
ZNot sure, but I know that when I started, I had a similar issue
Zmemory shot up
ZZtried a couple days later
Zworked out the box
ZOnly thing I remember is checking save state, so I can resume
Zin the first
THmm okay, I'll make sure I'm not using it.. I was thinking to also resume, since this is on a slow GPU. But I'm confident i've been testing with clean configs.. just change model and training folders.
Zand that was even on 64GB ram
Zso I know that was not the issue, lol
TSo for you it just started working? My was working, then i upgraded and can't get it working again
Zright, for me, no update in between. just a day or two, different instance, no changes
Zin my behavior, except checking that save state
Z(at least I'd like to think that was it)
TI have been using it on my other machine with 3090 but will for sure keep it off while i try to get this resolved on my older GPU
Zhaha, yeah, I moved training to Massed Compute
Zbecause couldn't stand the thought of blocking my GPU
Zfollowed once comment, do 140 epochs, then take that and do 60 more
ZI tricked my ex-wife
Z(I trained model on one of my kids)
Tyeah i had a 3090 for gaming, and now that I'm into AI, its cut into my gaming hobby.
Zyeah, used to be 80-150h/week in TFD or D2, now, nothing for the past two weeks
Zhaha
Zmaybe every two weeks
Z(these numbers seem high, lol)
Z
Tthats why I'm trying to get my 2nd GPU running.. just let that run all the time and free up primary PC.. plus i want to actually generate images and use all the cool AI stuff, but you also can't do that whie training.
Zessentially, fully train on 1 GPU on MC is 24-28h
Zso $10 a pop
Zworth for me
Zinstead of 5 days on home gpu
ZBut I get it
Tits taken me 6 versions to get my own likeness perfect.
LI have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_IT.json to train LORA? I had posted 8.94s with Doc's V9, but unfortunately after some times the training process failed due to lack of VRAM. Now I'm testing with Kohya original version and it reached 10.5 s/It. But it is more stable in VRAM usage.
C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:
with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]
C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning:
torch.cpu.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cpu', args...) instead.with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]
ZYou should try the suggested approach. 140 Epochs (1 iteration per image) on dev model
Zthen take the 140th epoch
Zand take that as input model
Zand train 60 more epochs with saving every 10th
Zit is incredible
TZI was
Tyeah i generally find in a 200 epoch run, the best is somewhere between 120-160, but never the 200's or less than 100
ZDreambooth vs Lora, no question 