Hi Dr, I'm trying to train a flux checkpoint with your 16GB config file but this error repeats every
Hi Dr, I'm trying to train a flux checkpoint with your 16GB config file but this error repeats every try and I can't figure out what is happening:
Fwhat error?
PTraceback (most recent call last):
File "Q:\kohya_ss\sd-scripts\train_db.py", line 529, in <module>
train(args)
File "Q:\kohya_ss\sd-scripts\train_db.py", line 123, in train
text_encoder, vae, unet, load_stable_diffusion_format = train_util.load_target_model(args, weight_dtype, accelerator)
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4825, in load_target_model
text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4780, in _load_target_model
text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 1005, in load_models_from_stable_diffusion_checkpoint
converted_unet_checkpoint = convert_ldm_unet_checkpoint(v2, state_dict, unet_config)
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 267, in convert_ldm_unet_checkpoint
new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
File "Q:\kohya_ss\sd-scripts\train_db.py", line 529, in <module>
train(args)
File "Q:\kohya_ss\sd-scripts\train_db.py", line 123, in train
text_encoder, vae, unet, load_stable_diffusion_format = train_util.load_target_model(args, weight_dtype, accelerator)
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4825, in load_target_model
text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "Q:\kohya_ss\sd-scripts\library\train_util.py", line 4780, in _load_target_model
text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 1005, in load_models_from_stable_diffusion_checkpoint
converted_unet_checkpoint = convert_ldm_unet_checkpoint(v2, state_dict, unet_config)
File "Q:\kohya_ss\sd-scripts\library\model_util.py", line 267, in convert_ldm_unet_checkpoint
new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
PKeyError: 'time_embed.0.weight'
Traceback (most recent call last):
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "Q:\kohya_ss\venv\Scripts\accelerate.EXE__main__.py", line 7, in <module>
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['Q:\kohya_ss\venv\Scripts\python.exe', 'Q:/kohya_ss/sd-scripts/train_db.py', '--config_file', 'Q://config_dreambooth-20241007-122824.toml']' returned non-zero exit status 1.
Traceback (most recent call last):
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Marcos\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "Q:\kohya_ss\venv\Scripts\accelerate.EXE__main__.py", line 7, in <module>
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "Q:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['Q:\kohya_ss\venv\Scripts\python.exe', 'Q:/kohya_ss/sd-scripts/train_db.py', '--config_file', 'Q://config_dreambooth-20241007-122824.toml']' returned non-zero exit status 1.
Fincorrect model
Fuse our model downloader
Fuse downloaded models
PHow many repeats per image in this config? with 50 images database
FRepeat always 1 for flux
Pty
SWhat's the practical effect of training a finetune rather than a lora? At the moment I have about 15 loras running in the dev checkpoint. If I run a finetune instead, that can only incorporate 1 lora within it and bloats the size of the file? Or am I just confused about how it works? Can I merge all my loras (they are all character loras) into 1 finetune?
FFine tuning is better quality simply
FLesser overfit
FWith same dataset
FAlso fine tuning is used to make a better general model too
FBut flux failing atm since it is distilled
Sbut finetuning leads to a file that is 10gig+ and lora leads to a 2gig file. I don't see the point of finetuning tbh
Fas i said better quality
Faim matters
Sfor something that is 10 times the size of the lora, I would expect it to be making me breakfast as well as being better quality :kekw:
Dfor some reason i can't get my shared vram above 24gb
D
Di have even tried the regedit for it
Fyou shouldnt use vram duıring training
F24 gb sufficient
Fwhich config u trying?
D12gb
Fok it should work what is the error?
Fyou can also try 10 gb and 8 gb and see how they perform
Di will test them
Fmonitor your vram before start and during training
Fyou can use
Fnvitop
Fpip install nvitop
Fnvitop
Dthe error message is cuda: out of memory Cuda kernal errors might be asynchronously reported at some other API call
Dthis happens as it starts the first step, so i assume that's the model load
PDr. I download the model and the other files with your script to download models and the error repeats every try, I indicate the folder of the model in Kohya.
PThe error is the same as above
Dthis is the complete error. it crashes at first step
D10gb json same error
D
Dnvitop shows it goes passed 48gb in loading and then crashes
Dit seems to crash when this section gets to more than 48gb
D
Dthe bottom mem number
Dmaybe my pagefile isn't set right?
D48gb is my actual ram, so it's crashing when it needs to go virtual maybe?
D@Dr. Furkan Gözükara i've ran some test. it seems to crash when physical memory is depleted, even though it has virtual memory left. Both are going down steadily, but once i run at out physical memory it crashes.
D
Dthis is the last second before the crash.