Fine tuning is better quality simply
Fine tuning is better quality simply
FLesser overfit
FWith same dataset
FAlso fine tuning is used to make a better general model too
FBut flux failing atm since it is distilled
Sbut finetuning leads to a file that is 10gig+ and lora leads to a 2gig file. I don't see the point of finetuning tbh
Fas i said better quality
Faim matters
Sfor something that is 10 times the size of the lora, I would expect it to be making me breakfast as well as being better quality :kekw:
Dfor some reason i can't get my shared vram above 24gb
D
Di have even tried the regedit for it
Fyou shouldnt use vram duıring training
F24 gb sufficient
Fwhich config u trying?
D12gb
Fok it should work what is the error?
Fyou can also try 10 gb and 8 gb and see how they perform
Di will test them
Fmonitor your vram before start and during training
Fyou can use
Fnvitop
Fpip install nvitop
Fnvitop
Dthe error message is cuda: out of memory Cuda kernal errors might be asynchronously reported at some other API call
Dthis happens as it starts the first step, so i assume that's the model load
PDr. I download the model and the other files with your script to download models and the error repeats every try, I indicate the folder of the model in Kohya.
PThe error is the same as above
Dthis is the complete error. it crashes at first step
D10gb json same error
D
Dnvitop shows it goes passed 48gb in loading and then crashes
Dit seems to crash when this section gets to more than 48gb
D
Dthe bottom mem number
Dmaybe my pagefile isn't set right?
D48gb is my actual ram, so it's crashing when it needs to go virtual maybe?
D@Dr. Furkan Gözükara i've ran some test. it seems to crash when physical memory is depleted, even though it has virtual memory left. Both are going down steadily, but once i run at out physical memory it crashes.
D
Dthis is the last second before the crash.
PDr Furkan I download the model and the other files with your script (Windows_Download_Training_Model_Files) and the error repeats in every try. I did a fresh install of Kohya ss and nothing that I did solve this error:
PTraceback (most recent call last):
File "Q:\kohya_ss\sd-scripts\train_db.py", line 529, in <module>
train(args)
File "Q:\kohya_ss\sd-scripts\train_db.py", line 123, in train
text_encoder, vae, unet, load_stable_diffusion_format = train_util.load_target_model(args, weight_dtype, accelerator)
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4825, in load_target_model
text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4780, in _load_target_model
text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 1005, in load_models_from_stable_diffusion_checkpoint
converted_unet_checkpoint = convert_ldm_unet_checkpoint(v2, state_dict, unet_config)
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 267, in convert_ldm_unet_checkpoint
new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
File "Q:\kohya_ss\sd-scripts\train_db.py", line 529, in <module>
train(args)
File "Q:\kohya_ss\sd-scripts\train_db.py", line 123, in train
text_encoder, vae, unet, load_stable_diffusion_format = train_util.load_target_model(args, weight_dtype, accelerator)
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4825, in load_target_model
text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4780, in _load_target_model
text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 1005, in load_models_from_stable_diffusion_checkpoint
converted_unet_checkpoint = convert_ldm_unet_checkpoint(v2, state_dict, unet_config)
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 267, in convert_ldm_unet_checkpoint
new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
PKeyError: 'time_embed.0.weight'
Traceback (most recent call last):
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "Q:\kohya_ss\venv\Scripts\accelerate.EXEmain.py", line 7, in <module>
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['Q:\kohya_ss\venv\Scripts\python.exe', 'Q:/kohya_ss/sd-scripts/train_db.py', '--config_file', 'Q://config_dreambooth-20241007-122824.toml']' returned non-zero exit status 1.
Traceback (most recent call last):
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "Q:\kohya_ss\venv\Scripts\accelerate.EXEmain.py", line 7, in <module>
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['Q:\kohya_ss\venv\Scripts\python.exe', 'Q:/kohya_ss/sd-scripts/train_db.py', '--config_file', 'Q://config_dreambooth-20241007-122824.toml']' returned non-zero exit status 1.
DIts impossible to set up a shared checkpoint and lora folder for different apps, I give up
Fabsolutely something wrong
Fi am using on RTX 3060 let me show
Feven 28 gb config works :d
F
Fyour shared vram must be off
Fi got 64 gb ram, you have 48 i think should work
Fcan you reduce your RAM usage?