Software Engineering Courses (SECourses)•15mo ago

IDK, it keeps going over 32.2GB then it gives out of memory, on SPDA. I'll keep trying other thin

IDK, it keeps going over 32.2GB then it gives out of memory, on SPDA. I'll keep trying other things.

theJindesanOP•10/26/24, 12:36 AM

Its like its not following the 35 blocks swap

TtheJindesan IDK, it keeps going over 32.2GB then it gives out of memory, on SPDA. I'll ke...

Zet•10/26/24, 2:19 AM

You didn't enable save state for some reason, right?

ZZet You didn't enable save state for some reason, right?

theJindesanOP•10/26/24, 2:26 AM

I have in some configs, will that use more memory just to save a state? But i was also testing clean configs from the v9 zip

Zet•10/26/24, 2:46 AM

Not sure, but I know that when I started, I had a similar issue

Zet•10/26/24, 2:46 AM

memory shot up

Zet•10/26/24, 2:46 AM

even talked to @Furkan Gözükara SECourses about it, no reason, no solution

Zet•10/26/24, 2:46 AM

tried a couple days later

Zet•10/26/24, 2:46 AM

worked out the box

Zet•10/26/24, 2:46 AM

Only thing I remember is checking save state, so I can resume

Zet•10/26/24, 2:47 AM

in the first

ZZet worked out the box

theJindesanOP•10/26/24, 2:47 AM

Hmm okay, I'll make sure I'm not using it.. I was thinking to also resume, since this is on a slow GPU. But I'm confident i've been testing with clean configs.. just change model and training folders.

Zet•10/26/24, 2:48 AM

and that was even on 64GB ram

Zet•10/26/24, 2:48 AM

so I know that was not the issue, lol

ZZet and that was even on 64GB ram

theJindesanOP•10/26/24, 2:49 AM

So for you it just started working? My was working, then i upgraded and can't get it working again

Zet•10/26/24, 2:49 AM

right, for me, no update in between. just a day or two, different instance, no changes

Zet•10/26/24, 2:49 AM

in my behavior, except checking that save state

Zet•10/26/24, 2:49 AM

(at least I'd like to think that was it)

ZZet (at least I'd like to think that was it)

theJindesanOP•10/26/24, 2:50 AM

I have been using it on my other machine with 3090 but will for sure keep it off while i try to get this resolved on my older GPU

Zet•10/26/24, 2:50 AM

haha, yeah, I moved training to Massed Compute

Zet•10/26/24, 2:51 AM

because couldn't stand the thought of blocking my GPU

Zet•10/26/24, 2:51 AM

followed once comment, do 140 epochs, then take that and do 60 more

Zet•10/26/24, 2:51 AM

I tricked my ex-wife

Zet•10/26/24, 2:51 AM

(I trained model on one of my kids)

ZZet because couldn't stand the thought of blocking my GPU

theJindesanOP•10/26/24, 2:51 AM

yeah i had a 3090 for gaming, and now that I'm into AI, its cut into my gaming hobby.

Zet•10/26/24, 2:52 AM

yeah, used to be 80-150h/week in TFD or D2, now, nothing for the past two weeks

Zet•10/26/24, 2:52 AM

haha

Zet•10/26/24, 2:52 AM

maybe every two weeks

Zet•10/26/24, 2:52 AM

(these numbers seem high, lol)

Zet•10/26/24, 2:52 AM

theJindesanOP•10/26/24, 2:52 AM

thats why I'm trying to get my 2nd GPU running.. just let that run all the time and free up primary PC.. plus i want to actually generate images and use all the cool AI stuff, but you also can't do that whie training.

Zet•10/26/24, 2:53 AM

essentially, fully train on 1 GPU on MC is 24-28h

Zet•10/26/24, 2:53 AM

so $10 a pop

Zet•10/26/24, 2:53 AM

worth for me

Zet•10/26/24, 2:54 AM

instead of 5 days on home gpu

Zet•10/26/24, 2:54 AM

But I get it

theJindesanOP•10/26/24, 2:55 AM

its taken me 6 versions to get my own likeness perfect.

TtheJindesan <@1141070068344688660> are you on 3090 or 3090 ti? are you overclocking? I onl...

Leolis78•10/26/24, 2:55 AM

I have Gigabyte Eagle rtx3090, no overclocked. Are you running Rank_3_T5_XXL_23500MB_11_35_Second_IT.json to train LORA? I had posted 8.94s with Doc's V9, but unfortunately after some times the training process failed due to lack of VRAM. Now I'm testing with Kohya original version and it reached 10.5 s/It. But it is more stable in VRAM usage.

C:\IA\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:295: FutureWarning: torch.cpu.amp.autocast(args...)torch.cpu.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cpu', args...)torch.amp.autocast('cpu', args...) instead.
with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined]
steps: 0%|▎ | 15/3000 [02:36<8:38:56, 10.43s/it, avr_loss=0.416]
epoch 2/200
2024-10-25 23:28:24 INFO epoch is incremented. current_epoch: 1, epoch: 2 train_util.py:715
steps: 1%|▌ | 30/3000 [05:11<8:34:22, 10.39s/it, avr_loss=0.406]
epoch 3/200
2024-10-25 23:31:00 INFO epoch is incremented. current_epoch: 2, epoch: 3 train_util.py:715
steps: 2%|▊ | 45/3000 [07:49<8:33:58, 10.44s/it, avr_loss=0.374]
epoch 4/200
2024-10-25 23:33:38 INFO epoch is incremented. current_epoch: 3, epoch: 4 train_util.py:715
steps: 2%|█ | 60/3000 [10:28<8:32:55, 10.47s/it, avr_loss=0.409]

TtheJindesan its taken me 6 versions to get my own likeness perfect.

Zet•10/26/24, 2:56 AM

You should try the suggested approach. 140 Epochs (1 iteration per image) on dev model

Zet•10/26/24, 2:56 AM

then take the 140th epoch

Zet•10/26/24, 2:56 AM

and take that as input model

Zet•10/26/24, 2:56 AM

and train 60 more epochs with saving every 10th

Zet•10/26/24, 2:56 AM

it is incredible

theJindesanOP•10/26/24, 2:56 AM

@Leolis78 ah, i thought you were doing the dreambooth method.. I do get faster lora training/it.

Zet•10/26/24, 2:57 AM

I was

ZZet then take the 140th epoch

theJindesanOP•10/26/24, 2:57 AM

yeah i generally find in a 200 epoch run, the best is somewhere between 120-160, but never the 200's or less than 100

Zet•10/26/24, 2:57 AM

Dreambooth vs Lora, no question

TtheJindesan yeah i generally find in a 200 epoch run, the best is somewhere between 120-160,...

Zet•10/26/24, 2:58 AM

Right, when you take that sweet spot at 140, and use that as your base model for training for an additional 60 epochs

Zet•10/26/24, 2:58 AM

with same dataset

theJindesanOP•10/26/24, 2:59 AM

@Zet do you happen to know what it means to be overtrained/overcooked. I see this term all the time in reference to too many epoc/repeats, but what does it actually mean? is the 200th epoch usually overtrained?

Zet•10/26/24, 3:00 AM

overtrained means that you see it converge towards likeness, then go away from it

IDK, it keeps going over 32.2GB then it gives out of memory, on SPDA. I'll keep trying other thin

Similar Threads