wait if TE is not trainable, I don't need to caption anymore?
wait if TE is not trainable, I don't need to caption anymore?
Fno you need to caption
Fcaptions are encoded and cached and used
U
�Hey @Furkan Gözükara SECourses . Been trying training in Runpod, I went with the pytorch 2.2 as explained in runpod installation in kohya v16 script of yours. Twice in a row the training has stopped, first it saved epoch 10, then stopped at epoch 20. i started new training using my saved epoch 10 checkpoint, now it proceeded to epoch 50, then I got this same error: (weird, because I selected 400 GB harddisk):
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"
�The last file was only saved to 21 GB, so it didn't complete for some reason
Flooks to me like pod hdd failed
�happened again: maybe turn off memory efficient save?
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
Fyep turn it off
Fif you have sufficient ram
Fshouldnt be needed
�running the a40 GPU
�oh, you're talking normal RAM.
�
�says 50 GB RAM, that's a weird amount
�ok, disabled memory efficient save now, continuing from last created epoch
Fyes system ram
MIf I may, be careful how many times you save intermediate [models, not modules] and states; I ended up with a full HD very soon. Remember that when Kohya save a training state (those folders whose name ends with 'state'), there's also a partially-trained, but full-size model in it. If it's a Flux-dev finetune, each file is 21 GB IIRC…
TDo you mean that if I need to resume, so I have to uncheck "memory efficient save" and set "Save last N epochs state" to 2 (if I want to keep the last 2 epoch ? (I don't really understand this setting, if I keep 200 epoch, save every 25 epoch and save last 2 epoch, is that mean that every epoch, the state is saved but keeping only 2 last ? or it keep only the epoch number 175 and 200 (state) ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?
�instead of setting flux dev as base, you set i.e. mycheckpoint-000075.safetensors. I would now rename the model to mycheckpoint-from75 so you remember where you started, then rename the numbers after by adding 75 
Tin this case, do you need to set also "save state with checkpoint" ?
�everything else is the same as before
�@Furkan Gözükara SECourses Is it possible to have 1.jpg 1.txt 2.jpg 2.txt while the rest of the pics are not manually captioned?
�ChatGPT says yes
RWhen it comes to training speed/performance, is there a noticiable difference between dreambooth and LoRA? Is the latter significantly faster? Or is the main the difference the size of the output?
Fcan I train a person and and object in the same training without captions?
Fit reads folder name if not finds .txt
Factually if you go with best quality
Ffine tuning faster than best of lora
Fhave them in different folders under the same main folder which you give as path
Fbut how good it will be i cant say for sure
FAnd how I make the trigger words?
Fohwx man
Fbbuk item
Fwhatever the item class
FBut there is two objects
Fthey will bleed
Fsmae class bleeds hard
FWell
Thank you
I’ll try !!
Thanks for your awesome work
Thank you
I’ll try !!
Thanks for your awesome work
Y@Furkan Gözükara SECourses I saw you mentioned you were doing research on the bleed problem, I was wondering if there was any new information?
YI have a Flux fine-tuned on a person, and I'm trying to pair with a LoRA trained on clothing. The face is getting really messed up unless I turn off the last lora block, and then the clothing lora isn't nearly as good, but the face works.
YI've now cropped out the faces in my clothing LoRA dataset, hoping that'll help
Fthey didnt work
YDo you think only training specific LoRA blocks would help me, just cropping the faces out?
Fi tested it
Fit doesnt help
Fcertain blocks
Feven single layer training still turns entire model into your face :d
