Software Engineering Courses (SECourses)•16mo ago

Hi <@205854764540362752> is there a way to save the training process into local computer then resume

Hi @Furkan Gözükara SECourses is there a way to save the training process into local computer then resume later on mass compute or runpod or something else. Due to training times is very long in case of server restart or crash, are there anyway to save the on going process then resume later ?

Wwardensc2 Hi <@205854764540362752> is there a way to save the training process into local ...

Furkan Gözükara SECourses•9/18/24, 5:13 PM

for fine tuning yes

Furkan Gözükara SECourses•9/18/24, 5:13 PM

use your latest checkpoint

FFurkan Gözükara SECourses it means that your initial vram usage was a lot

AstraBat•9/19/24, 12:37 AM

'block swaps' - I presume that's somewhere in the kohya_ss gui, specifically "double_blocks_to_swap = 5 (in the .toml file)?

AstraBat•9/19/24, 12:44 AM

I kept my VRAM right on 23.9 GB using the "Quality_1_23100MB_14_12_Second_IT.json" config (on a 24 GB card). I had to try a few tricks to reduce VRAM usage, such as switching off hardware accelerated GPU scheduling, and selecting 'Adjust for best performance' under Visual Effects in the Control Panel for Advanced System Settings. This helped a little though I still had 400 MB shared on RAM.

AAstraBat I kept my VRAM right on 23.9 GB using the "Quality_1_23100MB_14_12_Second_IT.js...

Furkan Gözükara SECourses•9/19/24, 1:02 AM

awesome

Furkan Gözükara SECourses•9/19/24, 1:02 AM

you can also compare swap 6 7 speed difference

Furkan Gözükara SECourses•9/19/24, 1:02 AM

and use fastest one

Furkan Gözükara SECourses•9/19/24, 1:02 AM

all quality same

AstraBat•9/19/24, 1:07 AM

I had tried 4e-06 learning rate, on 26 input images, but all of my epoch results looked similar (I was saving checkpoints every tenth epoch), so I tried again on 5e-06 LR and saw some improvement, though very gradual. I also noticed, right from my earliest checkpoint save, that sample photographic images of a 'woman' looked identical to those for 'ohxw woman', whereas for SDXL dreambooth the two images converged once the model was being over-trained in the new subject.

AAstraBat I had tried 4e-06 learning rate, on 26 input images, but all of my epoch results...

Furkan Gözükara SECourses•9/19/24, 1:18 AM

it is true that 05 06 or 03 may work better depending on dataset

Furkan Gözükara SECourses•9/19/24, 1:18 AM

really hard to set one LR for all

Furkan Gözükara SECourses•9/19/24, 1:18 AM

i tried to set an LR that will not very undertrain or overtrain

Furkan Gözükara SECourses•9/19/24, 1:19 AM

but with my LR as you do more training, you should eventually get your perfect checkpoint

FFurkan Gözükara SECourses but with my LR as you do more training, you should eventually get your perfect c...

AstraBat•9/19/24, 1:21 AM

Yes, I eventually got a good result.

FFurkan Gözükara SECourses for fine tuning yes

wardensc2OP•9/19/24, 2:31 AM

Can you describe more out which step to do it. Where I can load the latest checkpoint and how to calculate the remain step for the latest checkpoint to continue

dxqb•9/19/24, 5:41 AM

I've seen Flux not really converge on realism when the LR is too low, no matter how long you run it. It changes facial features but remains in its plastic AI style with a low LR

dxqb•9/19/24, 5:42 AM

are the e-06 values above with rank 128 alpha 16?

Wwardensc2 Can you describe more out which step to do it. Where I can load the latest check...

Furkan Gözükara SECourses•9/19/24, 10:54 AM

it wont have those

Furkan Gözükara SECourses•9/19/24, 10:54 AM

you need to memorize and do yourself

Ddxqb are the e-06 values above with rank 128 alpha 16?

Furkan Gözükara SECourses•9/19/24, 10:55 AM

dont change alpha we have it affects LR

dxqb•9/19/24, 11:41 AM

I have it at 1 but different LRs, that's why I'm asking. Alpha 16 multiplies the LR by 16 compared to 1

Ddxqb I have it at 1 but different LRs, that's why I'm asking. Alpha 16 multiplies the...

Furkan Gözükara SECourses•9/19/24, 12:06 PM

rank 16 + alpha 16 you trying?

Furkan Gözükara SECourses•9/19/24, 12:06 PM

you should make them equal

Furkan Gözükara SECourses•9/19/24, 12:06 PM

or your LR need to be researched again

dxqb•9/19/24, 12:06 PM

rank 128 alpha 1

dxqb•9/19/24, 12:06 PM

they don't have to be equal

dxqb•9/19/24, 12:07 PM

alpha is there so you can change the rank without changing the LR

Ddxqb they don't have to be equal

Furkan Gözükara SECourses•9/19/24, 12:07 PM

then you need new LR

dxqb•9/19/24, 12:07 PM

so whatever alpha you use, don't change it. but it doesn't matter what alpha you use.

Furkan Gözükara SECourses•9/19/24, 12:07 PM

actually we make alpha equal to rank

Furkan Gözükara SECourses•9/19/24, 12:07 PM

so that you can use same LR

dxqb•9/19/24, 12:08 PM

it's the other way around. if you use alpha 16, you can change the rank without changing the LR. OT (and I think kohya) automatically scale the LR by (alpha / rank)

Furkan Gözükara SECourses•9/19/24, 12:08 PM

i recently researched this i am 100% sure

dxqb•9/19/24, 12:08 PM

just keep alpha the same

Furkan Gözükara SECourses•9/19/24, 12:08 PM

i even talked this with kohya

Furkan Gözükara SECourses•9/19/24, 12:08 PM

alpha is just a multiplier that is used to balance LR for different lora ranks

dxqb•9/19/24, 12:08 PM

exactly. we mean the same thing