hey guys ! ANy tips on fastest way to copy my 46GB safetensor files from masscompute server to my lo
hey guys ! ANy tips on fastest way to copy my 46GB safetensor files from masscompute server to my local drive. I want to use in comfyui
Ci upload to hf or any cloud. then download them later
https://medium.com/@yushantripleseven/uploading-from-jupyter-to-huggingface-ccf352f1e049
https://medium.com/@yushantripleseven/uploading-from-jupyter-to-huggingface-ccf352f1e049
MHey Guys, Doctor… When training for a person or character (LoRA or finetune), what to do of training images where our subject is very well visible, nice resolution, nice lighting, etc… but… there's another person next to them?
1. Do not use the image for training
2. "Censor" the other person with a colored rectangle/circle using a color that is not in the general palette of the image
3. Crop the image, even if it means having a very weird form factor (e.g. very narrow and tall picture)
1. Do not use the image for training
2. "Censor" the other person with a colored rectangle/circle using a color that is not in the general palette of the image
3. Crop the image, even if it means having a very weird form factor (e.g. very narrow and tall picture)
Fhugging face
Fdont use
Fi dont use 2 person unless you are training 2 person at the same time
MThank you doc… May I also ask if you or anyone else here have ever resumed a training from Hugging Face in Kohya_SS? My training died as the disk went out of space, but I do have the saved states on HF
Fi never tried
MAnyways, I've been lucky enough to get a few credits on TensorDock and I've been trying your presets for Dreambooth finetuning on 1 and 2x H100's
MBasically, you get 1.68 s/it on 1 and 2.36 s/it on the 2x system [fixed it/s → s/it]
MCorrect me if I'm wrong but this is better-than-linear scaling, right? I mean, it takes far less than 2x 1.68s (3.36 s) to do a step
MBTW I don't know if it happens to you too, but Kohya_ss ends training with a bunch of errors even when everything went fine… Pretty scary…
Fno it didnt scale linearly
Fare the pci ex?
Fyou got like only 30% speed up
MI noticed VRAM load is disproportionately higher when running the finetune on the 2x GPU: 100% on both GPUs vs. 45% on the single system. [CPU → GPU]
MThat is a lot of overhead, possibly a bug in the script?
Fi reported this already
Fkohya doesnt know
Ffrom 28 gb to 79 gb it goes
MAh yes now I remember!
-My error remains.... https://www.youtube.com/watch?v=AY6DMBCIZ3A&t=400s ... I tried 19 times, but not easy to catch up.. sorry to my knoledge and my brain bad..... : ( and bad my english...
YouTubeSECourses
Updated for SDXL 1.0. How to install #Kohya SS GUI trainer and do #LoRA training with Stable Diffusion XL (#SDXL) this is the video you are looking for. I have shown how to install Kohya from scratch. The best parameters to do LoRA training with SDXL. How to use Kohya SDXL LoRAs with ComfyUI. How to do checkpoint comparison with SDXL LoRAs and m...
-message.txt3.44KB
Flooks like kohya install error
Fyou can use my this installer
Fit works with sdxl too
FPatreon
Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon
FYouTubeSECourses
If you are interested in using AI, generative AI applications, open source applications in your computer, than this is the most fundamental and important tutorial that you need. In this tutorial I show and explain how to properly install appropriate Python versions accurately, how to switch between different Python versions, how to install diffe...
Fif you need extra help i am giving private consultation
Y@Furkan Gözükara SECourses hi again, in the dreambooth training, why you not training the TE? bad results?
Fit doesnt work sadly
Fthey didnt add yet 
Yah damn ohwell my captions is natural language anyway so
Ythank u!
Fyou are welcome
Fsoon i will publish a video about improved clip l hopefully 
Fit will improve quality withno changes
Ywait if TE is not trainable, I don't need to caption anymore?
Fno you need to caption
Fcaptions are encoded and cached and used
U
�Hey @Furkan Gözükara SECourses . Been trying training in Runpod, I went with the pytorch 2.2 as explained in runpod installation in kohya v16 script of yours. Twice in a row the training has stopped, first it saved epoch 10, then stopped at epoch 20. i started new training using my saved epoch 10 checkpoint, now it proceeded to epoch 50, then I got this same error: (weird, because I selected 400 GB harddisk):
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"
�The last file was only saved to 21 GB, so it didn't complete for some reason
Flooks to me like pod hdd failed
�happened again: maybe turn off memory efficient save?
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
Fyep turn it off
Fif you have sufficient ram
Fshouldnt be needed
�running the a40 GPU
�oh, you're talking normal RAM.
�
