both are running now, I went with a different folder for the 2nd training, just diffusion_models2 to
both are running now, I went with a different folder for the 2nd training, just diffusion_models2 to be sure they don't get mixed up
οΏ½Running smoothly 
Fye use different image folders
For 1 overwrites others cache
οΏ½yeah, luckily I thought so before I started the 2nd one
οΏ½Uploaded checkpoints and pics using the huggingface script, but having all those pics in the same directory as the models became very annoying, how to I delete all pics from the huggingface directory?
Fsadly i dont know how to delete all try llms
οΏ½ok, I don't really need hugging for pics anyways, it's really for models
οΏ½Getting error using cropper on the newest version of ultimate img batch crop/resize, it can't save the files...using massedcompute. Write Error
Fon windows i didnt have errors
οΏ½one of the files had format jiff, might that be the reason for the error?
Fpngs working?
οΏ½I used an earlier version of your program on my pc instead of massed, it completed the process, and just ignored the jiff file
Fit is jiff or jfif?
Fi dont know jiff format
οΏ½looks like the new version doesnt ignore it, but stops the whole thing
οΏ½ah yeah, jfif probably
Fplease verify extension for me
Fso i can try to fix
οΏ½ok sec
οΏ½it is jfif
οΏ½there was also a webp pic in the folder, but that format works fine
Fon windows jfif works
Fi wonder if your image is broken
Fthe error is yolo not supporting
F'pfm', 'mpo', 'jpg', 'jpeg', 'webp', 'tif', 'dng', 'png', 'bmp', 'heic', 'tiff'
οΏ½ok thank you Dr.
Fi am gonna make a fix
οΏ½I am missing the ae.safetensors for flux dev, is this the right one? https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/ae.safetensors says flux schnell, but the vae is the same, ye?
οΏ½tried use the download models again, but it starts downloading flux dev etc. all over again
οΏ½Got error on start of training, I have not gotten this error before. done 20 trainings and no problems starting:
"024-12-09 12:23:35 INFO Loading state dict from flux_utils.py:330
/home/Ubuntu/Downloads/t5xxl_fp16
.safetensors
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 366, in load_safetensors
state_dict = load_file(path, device=device)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
with safe_open(filename, framework="pt", device=device) as f:
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 222, in train
t5xxl = flux_utils.load_t5xxl(args.t5xxl, weight_dtype, "cpu", args.disable_mmap_load_safetensors)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/flux_utils.py", line 331, in load_t5xxl
sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 368, in load_safetensors
state_dict = load_file(path) # prevent device invalid Error
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
"
"024-12-09 12:23:35 INFO Loading state dict from flux_utils.py:330
/home/Ubuntu/Downloads/t5xxl_fp16
.safetensors
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 366, in load_safetensors
state_dict = load_file(path, device=device)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
with safe_open(filename, framework="pt", device=device) as f:
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 222, in train
t5xxl = flux_utils.load_t5xxl(args.t5xxl, weight_dtype, "cpu", args.disable_mmap_load_safetensors)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/flux_utils.py", line 331, in load_t5xxl
sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 368, in load_safetensors
state_dict = load_file(path) # prevent device invalid Error
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
"
οΏ½found it I think, the t5xxl was 9.0 GB, that means it hadn't completed the download...
Ffixed batch pre processor
Fv8
Fmodel download fgailed
Fyes you found accurately
οΏ½Thank you Dr. Great work as always
οΏ½Can I use the option RTX A6000 NVLink with 2gpus and run 2 trainings at once?
Ffor fine tune i didnt test probably OOM
οΏ½OOM? it's 48 GB each
Fyes
Ffine tuning need 80 gb gpus
Fwhen you do multi gpu
οΏ½I'm talking two separate trainings
οΏ½on same VM
οΏ½I already did that with 2 a6000 before, so that works, but I'm unsure what the NVlink means
Fseperate works