Software Engineering Courses (SECourses)•13mo ago

<@205854764540362752> hi again, in the dreambooth training, why you not training the TE? bad results

@Furkan Gözükara SECourses hi again, in the dreambooth training, why you not training the TE? bad results?

YYobin <@205854764540362752> hi again, in the dreambooth training, why you not training...

Furkan Gözükara SECourses•12/12/24, 1:08 PM

it doesnt work sadly

Furkan Gözükara SECourses•12/12/24, 1:08 PM

they didnt add yet

FFurkan Gözükara SECourses it doesnt work sadly

YobinOP•12/12/24, 1:09 PM

ah damn ohwell my captions is natural language anyway so

YobinOP•12/12/24, 1:09 PM

thank u!

YYobin thank u!

Furkan Gözükara SECourses•12/12/24, 1:10 PM

you are welcome

Furkan Gözükara SECourses•12/12/24, 1:10 PM

soon i will publish a video about improved clip l hopefully

Furkan Gözükara SECourses•12/12/24, 1:10 PM

it will improve quality withno changes

FFurkan Gözükara SECourses you are welcome

YobinOP•12/12/24, 2:14 PM

wait if TE is not trainable, I don't need to caption anymore?

YYobin wait if TE is not trainable, I don't need to caption anymore?

Furkan Gözükara SECourses•12/12/24, 2:19 PM

no you need to caption

Furkan Gözükara SECourses•12/12/24, 2:19 PM

captions are encoded and cached and used

uisato•12/12/24, 2:48 PM

New short concept video! - https://www.youtube.com/watch?v=0cWBtiXhTTI&ab_channel=uisato

YouTubeuisato

Amok - Atoms for Peace [Short concept video / Custom FLUX model → "...

More experiments, through: https://linktr.ee/uisato

#design #animation #ai

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 5:56 PM

Hey @Furkan Gözükara SECourses . Been trying training in Runpod, I went with the pytorch 2.2 as explained in runpod installation in kohya v16 script of yours. Twice in a row the training has stopped, first it saved epoch 10, then stopped at epoch 20. i started new training using my saved epoch 10 checkpoint, now it proceeded to epoch 50, then I got this same error: (weird, because I selected 400 GB harddisk):
"aving checkpoint: /workspace/final_training_models/model/bodylora_head-000040.safetensors train_util.py:5664
Using memory efficient save file: /workspace/final_training_models/model/bodylora_head-000040.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)
[...]
mem_eff_save_file(state_dict, ckpt_path, metadata=sai_metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 260, in mem_eff_save_file
v.contiguous().view(torch.uint8).numpy().tofile(f)
OSError: 94371840 requested and 58861824 written
steps: 42%|████████████████████████████████████████████████████▌ | 200/472 [1:47:08<2:25:42, 32.14s/it, avr_loss=0.314]
Traceback (most recent call last):
File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
sys.exit(main())"

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 6:10 PM

The last file was only saved to 21 GB, so it didn't complete for some reason

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭Hey <@205854764540362752> . Been trying training in Runpod, I went with the pyto...

Furkan Gözükara SECourses•12/12/24, 7:38 PM

looks to me like pod hdd failed

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:42 PM

happened again: maybe turn off memory efficient save?

Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭happened again: maybe turn off memory efficient save? Using memory efficient sa...

Furkan Gözükara SECourses•12/12/24, 7:44 PM

yep turn it off

Furkan Gözükara SECourses•12/12/24, 7:44 PM

if you have sufficient ram

Furkan Gözükara SECourses•12/12/24, 7:44 PM

shouldnt be needed

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:44 PM

running the a40 GPU

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:44 PM

oh, you're talking normal RAM.

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:45 PM

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:45 PM

says 50 GB RAM, that's a weird amount

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/12/24, 7:49 PM

ok, disabled memory efficient save now, continuing from last created epoch

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok, disabled memory efficient save now, continuing from last created epoch

Furkan Gözükara SECourses•12/12/24, 8:01 PM

yes system ram

macba•12/12/24, 9:09 PM

If I may, be careful how many times you save intermediate [models, not modules] and states; I ended up with a full HD very soon. Remember that when Kohya save a training state (those folders whose name ends with 'state'), there's also a partially-trained, but full-size model in it. If it's a Flux-dev finetune, each file is 21 GB IIRC…

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok, disabled memory efficient save now, continuing from last created epoch

Testitou•12/13/24, 8:52 AM

Do you mean that if I need to resume, so I have to uncheck "memory efficient save" and set "Save last N epochs state" to 2 (if I want to keep the last 2 epoch ? (I don't really understand this setting, if I keep 200 epoch, save every 25 epoch and save last 2 epoch, is that mean that every epoch, the state is saved but keeping only 2 last ? or it keep only the epoch number 175 and 200 (state) ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/13/24, 8:56 AM

instead of setting flux dev as base, you set i.e. mycheckpoint-000075.safetensors. I would now rename the model to mycheckpoint-from75 so you remember where you started, then rename the numbers after by adding 75

Testitou•12/13/24, 8:58 AM

in this case, do you need to set also "save state with checkpoint" ?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/13/24, 9:05 AM

everything else is the same as before

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/13/24, 9:06 AM

@Furkan Gözükara SECourses Is it possible to have 1.jpg 1.txt 2.jpg 2.txt while the rest of the pics are not manually captioned?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭•12/13/24, 10:07 AM

ChatGPT says yes

Renzo•12/13/24, 6:06 PM

When it comes to training speed/performance, is there a noticiable difference between dreambooth and LoRA? Is the latter significantly faster? Or is the main the difference the size of the output?