Software Engineering Courses (SECourses)•13mo ago

Thank you doc… May I also ask if you or anyone else here have ever resumed a training from Hugging F

Thank you doc… May I also ask if you or anyone else here have ever resumed a training from Hugging Face in Kohya_SS? My training died as the disk went out of space, but I do have the saved states on HF

Mmacba Thank you doc… May I also ask if you or anyone else here have ever resumed a tra...

Furkan Gözükara SECourses•12/11/24, 8:20 PM

i never tried

macbaOP•12/11/24, 8:44 PM

Anyways, I've been lucky enough to get a few credits on TensorDock and I've been trying your presets for Dreambooth finetuning on 1 and 2x H100's

macbaOP•12/11/24, 8:46 PM

Basically, you get 1.68 s/it on 1 and 2.36 s/it on the 2x system [fixed it/s → s/it]

macbaOP•12/11/24, 8:46 PM

Correct me if I'm wrong but this is better-than-linear scaling, right? I mean, it takes far less than 2x 1.68s (3.36 s) to do a step

macbaOP•12/11/24, 8:47 PM

BTW I don't know if it happens to you too, but Kohya_ss ends training with a bunch of errors even when everything went fine… Pretty scary…

Mmacba Basically, you get 1.68 s/it on 1 and 2.36 s/it on the 2x system [fixed it/s → s...

Furkan Gözükara SECourses•12/11/24, 9:01 PM

no it didnt scale linearly

Furkan Gözükara SECourses•12/11/24, 9:01 PM

are the pci ex?

Furkan Gözükara SECourses•12/11/24, 9:02 PM

you got like only 30% speed up

macbaOP•12/11/24, 9:52 PM

I noticed VRAM load is disproportionately higher when running the finetune on the 2x GPU: 100% on both GPUs vs. 45% on the single system. [CPU → GPU]

macbaOP•12/11/24, 9:53 PM

That is a lot of overhead, possibly a bug in the script?

Mmacba I noticed VRAM load is disproportionately higher when running the finetune on th...

Furkan Gözükara SECourses•12/11/24, 9:55 PM

i reported this already

Furkan Gözükara SECourses•12/11/24, 9:55 PM

kohya doesnt know

Furkan Gözükara SECourses•12/11/24, 9:55 PM

from 28 gb to 79 gb it goes

macbaOP•12/11/24, 9:56 PM

Ah yes now I remember!

-•12/12/24, 6:00 AM

My error remains.... https://www.youtube.com/watch?v=AY6DMBCIZ3A&t=400s ... I tried 19 times, but not easy to catch up.. sorry to my knoledge and my brain bad..... : ( and bad my english...

YouTubeSECourses

First Ever SDXL Training With Kohya LoRA - Stable Diffusion XL Trai...

Updated for SDXL 1.0. How to install #Kohya SS GUI trainer and do #LoRA training with Stable Diffusion XL (#SDXL) this is the video you are looking for. I have shown how to install Kohya from scratch. The best parameters to do LoRA training with SDXL. How to use Kohya SDXL LoRAs with ComfyUI. How to do checkpoint comparison with SDXL LoRAs and m...

-•12/12/24, 6:02 AM

message.txt3.44KB

--My error remains.... https://www.youtube.com/watch?v=AY6DMBCIZ3A&t=400s ....

Furkan Gözükara SECourses•12/12/24, 11:25 AM

looks like kohya install error

Furkan Gözükara SECourses•12/12/24, 11:25 AM

you can use my this installer

Furkan Gözükara SECourses•12/12/24, 11:25 AM

it works with sdxl too

Furkan Gözükara SECourses•12/12/24, 11:26 AM

https://www.patreon.com/posts/112099700

Patreon

Kohya FLUX Fine Tuning (Full Checkpoints) Training Full Tutorial Fo...

Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon

Furkan Gözükara SECourses•12/12/24, 11:26 AM

also i recommend you to follow every step of this video first : https://youtu.be/DrhUHnYfwC0

YouTubeSECourses

How to Install Python, CUDA, cuDNN, C++ Build Tools, FFMPEG & Git T...

If you are interested in using AI, generative AI applications, open source applications in your computer, than this is the most fundamental and important tutorial that you need. In this tutorial I show and explain how to properly install appropriate Python versions accurately, how to switch between different Python versions, how to install diffe...

Furkan Gözükara SECourses•12/12/24, 11:26 AM

if you need extra help i am giving private consultation

Yobin•12/12/24, 1:05 PM

@Furkan Gözükara SECourses hi again, in the dreambooth training, why you not training the TE? bad results?

YYobin <@205854764540362752> hi again, in the dreambooth training, why you not training...

Furkan Gözükara SECourses•12/12/24, 1:08 PM

it doesnt work sadly

Furkan Gözükara SECourses•12/12/24, 1:08 PM

they didnt add yet

FFurkan Gözükara SECourses it doesnt work sadly

Yobin•12/12/24, 1:09 PM

ah damn ohwell my captions is natural language anyway so

Yobin•12/12/24, 1:09 PM

thank u!

YYobin thank u!

Furkan Gözükara SECourses•12/12/24, 1:10 PM

you are welcome

Furkan Gözükara SECourses•12/12/24, 1:10 PM

soon i will publish a video about improved clip l hopefully

Furkan Gözükara SECourses•12/12/24, 1:10 PM

it will improve quality withno changes

FFurkan Gözükara SECourses you are welcome

Yobin•12/12/24, 2:14 PM

wait if TE is not trainable, I don't need to caption anymore?

YYobin wait if TE is not trainable, I don't need to caption anymore?

Furkan Gözükara SECourses•12/12/24, 2:19 PM

no you need to caption

Furkan Gözükara SECourses•12/12/24, 2:19 PM

captions are encoded and cached and used

uisato•12/12/24, 2:48 PM

New short concept video! - https://www.youtube.com/watch?v=0cWBtiXhTTI&ab_channel=uisato

YouTubeuisato

Amok - Atoms for Peace [Short concept video / Custom FLUX model → "...

More experiments, through: https://linktr.ee/uisato

#design #animation #ai

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 5:56 PM

Hey @Furkan Gözükara SECourses . Been trying training in Runpod, I went with the pytorch 2.2 as explained in runpod installation in kohya v16 script of yours. Twice in a row the training has stopped, first it saved epoch 10, then stopped at epoch 20. i started new training using my saved epoch 10 checkpoint, now it proceeded to epoch 50, then I got this same error: (weird, because I selected 400 GB harddisk):
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 6:10 PM

The last file was only saved to 21 GB, so it didn't complete for some reason

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭Hey <@205854764540362752> . Been trying training in Runpod, I went with the pyto...

Furkan Gözükara SECourses•12/12/24, 7:38 PM

looks to me like pod hdd failed

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:42 PM

happened again: maybe turn off memory efficient save?

Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭happened again: maybe turn off memory efficient save? Using memory efficient sa...

Furkan Gözükara SECourses•12/12/24, 7:44 PM

yep turn it off

Furkan Gözükara SECourses•12/12/24, 7:44 PM

if you have sufficient ram

Furkan Gözükara SECourses•12/12/24, 7:44 PM

shouldnt be needed

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:44 PM

running the a40 GPU

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:44 PM

oh, you're talking normal RAM.

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:45 PM

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:45 PM

says 50 GB RAM, that's a weird amount

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:49 PM

ok, disabled memory efficient save now, continuing from last created epoch

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok, disabled memory efficient save now, continuing from last created epoch

Furkan Gözükara SECourses•12/12/24, 8:01 PM

yes system ram

macbaOP•12/12/24, 9:09 PM

If I may, be careful how many times you save intermediate [models, not modules] and states; I ended up with a full HD very soon. Remember that when Kohya save a training state (those folders whose name ends with 'state'), there's also a partially-trained, but full-size model in it. If it's a Flux-dev finetune, each file is 21 GB IIRC…

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok, disabled memory efficient save now, continuing from last created epoch

Testitou•12/13/24, 8:52 AM

Do you mean that if I need to resume, so I have to uncheck "memory efficient save" and set "Save last N epochs state" to 2 (if I want to keep the last 2 epoch ? (I don't really understand this setting, if I keep 200 epoch, save every 25 epoch and save last 2 epoch, is that mean that every epoch, the state is saved but keeping only 2 last ? or it keep only the epoch number 175 and 200 (state) ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/13/24, 8:56 AM

instead of setting flux dev as base, you set i.e. mycheckpoint-000075.safetensors. I would now rename the model to mycheckpoint-from75 so you remember where you started, then rename the numbers after by adding 75

Thank you doc… May I also ask if you or anyone else here have ever resumed a training from Hugging F

Similar Threads

Similar Threads

Similar Threads