Fi am gonna make a fix
๏ฟฝI am missing the ae.safetensors for flux dev, is this the right one? https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/ae.safetensors says flux schnell, but the vae is the same, ye?
๏ฟฝtried use the download models again, but it starts downloading flux dev etc. all over again
๏ฟฝGot error on start of training, I have not gotten this error before. done 20 trainings and no problems starting:
"024-12-09 12:23:35 INFO Loading state dict from flux_utils.py:330
/home/Ubuntu/Downloads/t5xxl_fp16
.safetensors
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 366, in load_safetensors
state_dict = load_file(path, device=device)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
with safe_open(filename, framework="pt", device=device) as f:
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 222, in train
t5xxl = flux_utils.load_t5xxl(args.t5xxl, weight_dtype, "cpu", args.disable_mmap_load_safetensors)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/flux_utils.py", line 331, in load_t5xxl
sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 368, in load_safetensors
state_dict = load_file(path) # prevent device invalid Error
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
"
"024-12-09 12:23:35 INFO Loading state dict from flux_utils.py:330
/home/Ubuntu/Downloads/t5xxl_fp16
.safetensors
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 366, in load_safetensors
state_dict = load_file(path, device=device)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
with safe_open(filename, framework="pt", device=device) as f:
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 222, in train
t5xxl = flux_utils.load_t5xxl(args.t5xxl, weight_dtype, "cpu", args.disable_mmap_load_safetensors)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/flux_utils.py", line 331, in load_t5xxl
sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/utils.py", line 368, in load_safetensors
state_dict = load_file(path) # prevent device invalid Error
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/safetensors/torch.py", line 313, in load_file
"
๏ฟฝfound it I think, the t5xxl was 9.0 GB, that means it hadn't completed the download...
Ffixed batch pre processor
Fv8
Fmodel download fgailed
Fyes you found accurately
๏ฟฝThank you Dr. Great work as always 
๏ฟฝCan I use the option RTX A6000 NVLink with 2gpus and run 2 trainings at once?
Ffor fine tune i didnt test probably OOM
๏ฟฝOOM? it's 48 GB each
Fyes
Ffine tuning need 80 gb gpus
Fwhen you do multi gpu
๏ฟฝI'm talking two separate trainings
๏ฟฝon same VM
๏ฟฝI already did that with 2 a6000 before, so that works, but I'm unsure what the NVlink means
Fseperate works
Fnvlink used when multiple gpu does a thing together
๏ฟฝCHat GPT O1-preview gave this answer: "Yes, you can run the two GPUs separately, treating each as its own device. Even though your RTX A6000 GPUs are connected via NVLinkโwhich enables high-bandwidth peer-to-peer communication and memory access between themโthey still present themselves as two distinct GPU devices to the system and to frameworks like PyTorch or TensorFlow"
Fyep
๏ฟฝIf I have 1.jpg 1.txt 2.jpg 2.txt, the captions will automatically be read from the txt files in kohya, yes?
๏ฟฝis it possible to convert an sdxl lora to a flux lora?
ZCan you make a LoRA from a fine-tuned model? That is, extract "ohwx man" from it to run tests? Is there any comparison in quality?
FPatreon
Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon
๏ฟฝdoing sdxl training on kaggle free, if your computer goes to sleep, will that stop the training? 
Fkaggle page has to stay open
Fand selected
๏ฟฝoh wow, has to be active tab too? insane 
Fi would leave that way :d
Ahey guys ! ANy tips on fastest way to copy my 46GB safetensor files from masscompute server to my local drive. I want to use in comfyui
Ci upload to hf or any cloud. then download them later
https://medium.com/@yushantripleseven/uploading-from-jupyter-to-huggingface-ccf352f1e049
https://medium.com/@yushantripleseven/uploading-from-jupyter-to-huggingface-ccf352f1e049
MHey Guys, Doctorโฆ When training for a person or character (LoRA or finetune), what to do of training images where our subject is very well visible, nice resolution, nice lighting, etcโฆ butโฆ there's another person next to them?
1. Do not use the image for training
2. "Censor" the other person with a colored rectangle/circle using a color that is not in the general palette of the image
3. Crop the image, even if it means having a very weird form factor (e.g. very narrow and tall picture)
1. Do not use the image for training
2. "Censor" the other person with a colored rectangle/circle using a color that is not in the general palette of the image
3. Crop the image, even if it means having a very weird form factor (e.g. very narrow and tall picture)
Fhugging face
Fdont use
Fi dont use 2 person unless you are training 2 person at the same time
MThank you docโฆ May I also ask if you or anyone else here have ever resumed a training from Hugging Face in Kohya_SS? My training died as the disk went out of space, but I do have the saved states on HF
Fi never tried
MAnyways, I've been lucky enough to get a few credits on TensorDock and I've been trying your presets for Dreambooth finetuning on 1 and 2x H100's
MBasically, you get 1.68 s/it on 1 and 2.36 s/it on the 2x system [fixed it/s โ s/it]
MCorrect me if I'm wrong but this is better-than-linear scaling, right? I mean, it takes far less than 2x 1.68s (3.36 s) to do a step
MBTW I don't know if it happens to you too, but Kohya_ss ends training with a bunch of errors even when everything went fineโฆ Pretty scaryโฆ
