like say you have 2048x2048 images and you train at 1024x1024 with buckets
like say you have 2048x2048 images
and you train at 1024x1024 with buckets
and you train at 1024x1024 with buckets
Mhave you tried that
NI don't think he has used bucketing in any of his flux configs so far
�What would happen if he just went 4K all the way lol
Fi trained
F1280
F1536
Fmy images
BGetting this issue on training on runpod
BTraceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train_network.py", line 445, in <module>
trainer.train(args)
File "/workspace/kohya_ss/sd-scripts/train_network.py", line 344, in train
model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "/workspace/kohya_ss/sd-scripts/flux_train_network.py", line 85, in load_target_model
clip_l = flux_utils.load_clip_l(args.clip_l, weight_dtype, "cpu", disable_mmap=args.disable_mmap_load_safetensors)
File "/workspace/kohya_ss/sd-scripts/library/flux_utils.py", line 168, in load_clip_l
sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
File "/workspace/kohya_ss/sd-scripts/library/flux_utils.py", line 39, in load_safetensors
return load_file(path) # prevent device invalid Error
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
with safe_open(filename, framework="pt", device=device) as f:
FileNotFoundError: No such file or directory: "/home/Ubuntu/Downloads/clip_l.safetensors"
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', '/workspace/AM/model/config_lora-20240828-180637.toml']' returned non-zero exit status 1.
File "/workspace/kohya_ss/sd-scripts/flux_train_network.py", line 445, in <module>
trainer.train(args)
File "/workspace/kohya_ss/sd-scripts/train_network.py", line 344, in train
model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "/workspace/kohya_ss/sd-scripts/flux_train_network.py", line 85, in load_target_model
clip_l = flux_utils.load_clip_l(args.clip_l, weight_dtype, "cpu", disable_mmap=args.disable_mmap_load_safetensors)
File "/workspace/kohya_ss/sd-scripts/library/flux_utils.py", line 168, in load_clip_l
sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
File "/workspace/kohya_ss/sd-scripts/library/flux_utils.py", line 39, in load_safetensors
return load_file(path) # prevent device invalid Error
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
with safe_open(filename, framework="pt", device=device) as f:
FileNotFoundError: No such file or directory: "/home/Ubuntu/Downloads/clip_l.safetensors"
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', '/workspace/AM/model/config_lora-20240828-180637.toml']' returned non-zero exit status 1.
Fi didnt do bucketing
Fso both images and resolution gui was big
Fdownload file
Fgive its accurate path
Frunpod tutorial coming hopefully very soon editing windows tutorial
Bok great
Fso currently it will be like
F/workspace/clip_l.safetensors
F@B0nd
BOk thank you
Dnative windows, better but still slower than WSl
DI traying with a ridiculously high TE lr to see what's happens
Nhow much does LR affect speed? 1024x1024 images? really hoping I can get your speed now that i've done the dev torch update
DI think none
D1024x1024 images
N387 images... are you trying for a style lora?
D3 people but is because the folders have 3 repetitions 129 pictures in total
Nah nice
D1 epoch is actually 3 epochs
DNative windows but with Apply T5 Attention Mask off, allmost the same speed as linux
Nvery nice. i'm trying first style lora. 696 images lol. T5 attention mask is on because i joy captioned all the images. took the images down to 512 since it's such a huge dataset. getting 3.04s/it
Dnice! Lets se how 512 resolution affects the quality
DT5 attention mask affects speed big time
Dmay be it was why I was having 4.1 it/s yesterday on wsl, maybe T5 attention mask was disabled
Ni really want to stay up till 1am and play with it when it's done but i'll probably fall asleep lol
P@Dr. Furkan Gözükara hey 
on runpod when I deploy comfy ui template, Jupiter lab doesn’t allow me to open the checkpoint folder to add safetensors. Is this normal? I’m able to open the the other folders such as Lora etc
on runpod when I deploy comfy ui template, Jupiter lab doesn’t allow me to open the checkpoint folder to add safetensors. Is this normal? I’m able to open the the other folders such as Lora etc
Swhen you do , lora training, how do you, implement captions to the training?
Njust a text file with the same base name as the picture. if extension is .txt, kohya should detect and use by default.
NJoy Caption started every caption with "This photograph..." so I used Doc's caption editor app to batch replace with my token
MI'm a bit of a Pony. I managed to find the right parameters under Fooocus for my personal Loras, which I trained to PonyRealism 2.1. Unfortunately none of the realistic Pony models are enough on their own, so I'm happy with the presets below:
DNative windows 1024x1024 same speed as wsl, T5 attention mask disabled
Dthe new torch worked
DI eliminated the class from the captions, Lets see if the bleeding is gone.
D@Dr. Furkan Gözükara what lr do you recommend for TE and UNET or you are still testing?
DIs for training multiple subjects I want to increase resemblance and decrease bleeding, I removed the class from the captions
DCould someone tell me why I've been stuck at 0% for 2 hours without an error message with an RTX 4090
Dyou are doing dreambooth or lora?
DHi, thx you. I try to creat lora
Dwith Kohya
Dyou are in the lora tab?
Dyes