Software Engineering Courses (SECourses)•15mo ago

What's the practical effect of training a finetune rather than a lora? At the moment I have about 15

What's the practical effect of training a finetune rather than a lora? At the moment I have about 15 loras running in the dev checkpoint. If I run a finetune instead, that can only incorporate 1 lora within it and bloats the size of the file? Or am I just confused about how it works? Can I merge all my loras (they are all character loras) into 1 finetune?

SSharinel What's the practical effect of training a finetune rather than a lora? At the mo...

Furkan Gözükara SECourses•10/7/24, 3:39 PM

Fine tuning is better quality simply

Furkan Gözükara SECourses•10/7/24, 3:40 PM

Lesser overfit

Furkan Gözükara SECourses•10/7/24, 3:40 PM

With same dataset

Furkan Gözükara SECourses•10/7/24, 3:40 PM

Also fine tuning is used to make a better general model too

Furkan Gözükara SECourses•10/7/24, 3:40 PM

But flux failing atm since it is distilled

FFurkan Gözükara SECourses Fine tuning is better quality simply

SharinelOP•10/7/24, 3:42 PM

but finetuning leads to a file that is 10gig+ and lora leads to a 2gig file. I don't see the point of finetuning tbh

SSharinel but finetuning leads to a file that is 10gig+ and lora leads to a 2gig file. I d...

Furkan Gözükara SECourses•10/7/24, 3:55 PM

as i said better quality

Furkan Gözükara SECourses•10/7/24, 3:55 PM

aim matters

SharinelOP•10/7/24, 3:58 PM

for something that is 10 times the size of the lora, I would expect it to be making me breakfast as well as being better quality :kekw:

Darth Teethius•10/7/24, 3:59 PM

for some reason i can't get my shared vram above 24gb

Darth Teethius•10/7/24, 3:59 PM

i have even tried the regedit for it

DDarth Teethius Click to see attachment

Furkan Gözükara SECourses•10/7/24, 4:06 PM

you shouldnt use vram duıring training

Furkan Gözükara SECourses•10/7/24, 4:06 PM

24 gb sufficient

Furkan Gözükara SECourses•10/7/24, 4:06 PM

which config u trying?

Darth Teethius•10/7/24, 4:07 PM

12gb

DDarth Teethius 12gb

Furkan Gözükara SECourses•10/7/24, 4:09 PM

ok it should work what is the error?

Furkan Gözükara SECourses•10/7/24, 4:09 PM

you can also try 10 gb and 8 gb and see how they perform

Darth Teethius•10/7/24, 4:09 PM

i will test them

Furkan Gözükara SECourses•10/7/24, 4:09 PM

monitor your vram before start and during training

Furkan Gözükara SECourses•10/7/24, 4:09 PM

you can use

Furkan Gözükara SECourses•10/7/24, 4:09 PM

nvitop

Furkan Gözükara SECourses•10/7/24, 4:10 PM

pip install nvitop

Furkan Gözükara SECourses•10/7/24, 4:10 PM

nvitop

Darth Teethius•10/7/24, 4:10 PM

the error message is cuda: out of memory Cuda kernal errors might be asynchronously reported at some other API call

Darth Teethius•10/7/24, 4:11 PM

this happens as it starts the first step, so i assume that's the model load

pianogospel•10/7/24, 4:20 PM

Dr. I download the model and the other files with your script to download models and the error repeats every try, I indicate the folder of the model in Kohya.

pianogospel•10/7/24, 4:30 PM

The error is the same as above

Darth Teethius•10/7/24, 4:44 PM

this is the complete error. it crashes at first step

Darth Teethius•10/7/24, 4:44 PM

10gb json same error

Darth Teethius•10/7/24, 4:44 PM

FFurkan Gözükara SECourses monitor your vram before start and during training

Darth Teethius•10/7/24, 4:47 PM

nvitop shows it goes passed 48gb in loading and then crashes

Darth Teethius•10/7/24, 5:37 PM

it seems to crash when this section gets to more than 48gb

Darth Teethius•10/7/24, 5:37 PM

Darth Teethius•10/7/24, 5:37 PM

the bottom mem number

Darth Teethius•10/7/24, 5:37 PM

maybe my pagefile isn't set right?

Darth Teethius•10/7/24, 5:38 PM

48gb is my actual ram, so it's crashing when it needs to go virtual maybe?

Darth Teethius•10/7/24, 6:49 PM

@Dr. Furkan Gözükara i've ran some test. it seems to crash when physical memory is depleted, even though it has virtual memory left. Both are going down steadily, but once i run at out physical memory it crashes.

Darth Teethius•10/7/24, 6:49 PM

Darth Teethius•10/7/24, 6:50 PM

this is the last second before the crash.

pianogospel•10/7/24, 6:55 PM

Dr Furkan I download the model and the other files with your script (Windows_Download_Training_Model_Files) and the error repeats in every try. I did a fresh install of Kohya ss and nothing that I did solve this error:

pianogospel•10/7/24, 6:55 PM

Traceback (most recent call last):
File "Q:\kohya_ss\sd-scripts\train_db.py", line 529, in <module>
train(args)
File "Q:\kohya_ss\sd-scripts\train_db.py", line 123, in train
text_encoder, vae, unet, load_stable_diffusion_format = train_util.load_target_model(args, weight_dtype, accelerator)
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4825, in load_target_model
text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4780, in _load_target_model
text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 1005, in load_models_from_stable_diffusion_checkpoint
converted_unet_checkpoint = convert_ldm_unet_checkpoint(v2, state_dict, unet_config)
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 267, in convert_ldm_unet_checkpoint
new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]

pianogospel•10/7/24, 6:56 PM

KeyError: 'time_embed.0.weight'
Traceback (most recent call last):
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "Q:\kohya_ss\venv\Scripts\accelerate.EXEmain.py", line 7, in <module>
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['Q:\kohya_ss\venv\Scripts\python.exe', 'Q:/kohya_ss/sd-scripts/train_db.py', '--config_file', 'Q://config_dreambooth-20241007-122824.toml']' returned non-zero exit status 1.