Software Engineering Courses (SECourses)•13mo ago

happened again: maybe turn off memory efficient save? Using memory efficient save file: /workspace/

happened again: maybe turn off memory efficient save?

Using memory efficient save file: /workspace/final_training_models/model/bodylora_head_from40-000020.safetensors
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 849, in <module>
train(args)

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭happened again: maybe turn off memory efficient save? Using memory efficient sa...

Furkan Gözükara SECourses•12/12/24, 7:44 PM

yep turn it off

Furkan Gözükara SECourses•12/12/24, 7:44 PM

if you have sufficient ram

Furkan Gözükara SECourses•12/12/24, 7:44 PM

shouldnt be needed

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/12/24, 7:44 PM

running the a40 GPU

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/12/24, 7:44 PM

oh, you're talking normal RAM.

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/12/24, 7:45 PM

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/12/24, 7:45 PM

says 50 GB RAM, that's a weird amount

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/12/24, 7:49 PM

ok, disabled memory efficient save now, continuing from last created epoch

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok, disabled memory efficient save now, continuing from last created epoch

Furkan Gözükara SECourses•12/12/24, 8:01 PM

yes system ram

macba•12/12/24, 9:09 PM

If I may, be careful how many times you save intermediate [models, not modules] and states; I ended up with a full HD very soon. Remember that when Kohya save a training state (those folders whose name ends with 'state'), there's also a partially-trained, but full-size model in it. If it's a Flux-dev finetune, each file is 21 GB IIRC…

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭ok, disabled memory efficient save now, continuing from last created epoch

Testitou•12/13/24, 8:52 AM

Do you mean that if I need to resume, so I have to uncheck "memory efficient save" and set "Save last N epochs state" to 2 (if I want to keep the last 2 epoch ? (I don't really understand this setting, if I keep 200 epoch, save every 25 epoch and save last 2 epoch, is that mean that every epoch, the state is saved but keeping only 2 last ? or it keep only the epoch number 175 and 200 (state) ?
Also, what is the setting to only keep for example 4 last checkpoints corresponding epoch 125, 150, 175 and 200 ?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/13/24, 8:56 AM

instead of setting flux dev as base, you set i.e. mycheckpoint-000075.safetensors. I would now rename the model to mycheckpoint-from75 so you remember where you started, then rename the numbers after by adding 75

Testitou•12/13/24, 8:58 AM

in this case, do you need to set also "save state with checkpoint" ?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/13/24, 9:05 AM

everything else is the same as before

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/13/24, 9:06 AM

@Furkan Gözükara SECourses Is it possible to have 1.jpg 1.txt 2.jpg 2.txt while the rest of the pics are not manually captioned?

�

🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭OP•12/13/24, 10:07 AM

ChatGPT says yes

Renzo•12/13/24, 6:06 PM

When it comes to training speed/performance, is there a noticiable difference between dreambooth and LoRA? Is the latter significantly faster? Or is the main the difference the size of the output?

franc aleu•12/13/24, 7:20 PM

can I train a person and and object in the same training without captions?

�🍭🎀 𝒜𝓋𝒶 𝐹𝓇𝒾𝑔𝑔 🎀🍭<@205854764540362752> Is it possible to have 1.jpg 1.txt 2.jpg 2.txt while the r...

Furkan Gözükara SECourses•12/13/24, 7:46 PM

yep

Furkan Gözükara SECourses•12/13/24, 7:46 PM

it reads folder name if not finds .txt

RRenzo When it comes to training speed/performance, is there a noticiable difference be...

Furkan Gözükara SECourses•12/13/24, 7:47 PM

actually if you go with best quality

Furkan Gözükara SECourses•12/13/24, 7:47 PM

fine tuning faster than best of lora

Ffranc aleu can I train a person and and object in the same training without captions?

Furkan Gözükara SECourses•12/13/24, 7:47 PM

yep

Furkan Gözükara SECourses•12/13/24, 7:47 PM

have them in different folders under the same main folder which you give as path

Furkan Gözükara SECourses•12/13/24, 7:47 PM

but how good it will be i cant say for sure

FFurkan Gözükara SECourses have them in different folders under the same main folder which you give as path

franc aleu•12/13/24, 7:49 PM

And how I make the trigger words?

Ffranc aleu And how I make the trigger words?

Furkan Gözükara SECourses•12/13/24, 7:49 PM

ohwx man

Furkan Gözükara SECourses•12/13/24, 7:49 PM

bbuk item

Furkan Gözükara SECourses•12/13/24, 7:49 PM

whatever the item class

franc aleu•12/13/24, 7:50 PM

But there is two objects

Ffranc aleu But there is two objects

Furkan Gözükara SECourses•12/13/24, 7:50 PM

they will bleed

Furkan Gözükara SECourses•12/13/24, 7:50 PM

smae class bleeds hard

franc aleu•12/13/24, 7:51 PM

Well
Thank you
I’ll try !!
Thanks for your awesome work

Ffranc aleu Well Thank you I’ll try !! Thanks for your awesome work

Furkan Gözükara SECourses•12/13/24, 7:51 PM

yolkypalky•12/13/24, 8:06 PM

@Furkan Gözükara SECourses I saw you mentioned you were doing research on the bleed problem, I was wondering if there was any new information?

yolkypalky•12/13/24, 8:08 PM

I have a Flux fine-tuned on a person, and I'm trying to pair with a LoRA trained on clothing. The face is getting really messed up unless I turn off the last lora block, and then the clothing lora isn't nearly as good, but the face works.

yolkypalky•12/13/24, 8:09 PM

I've now cropped out the faces in my clothing LoRA dataset, hoping that'll help

Yyolkypalky <@205854764540362752> I saw you mentioned you were doing research on the bleed ...

Furkan Gözükara SECourses•12/13/24, 8:10 PM

they didnt work

yolkypalky•12/13/24, 8:13 PM

Do you think only training specific LoRA blocks would help me, just cropping the faces out?

Yyolkypalky Do you think only training specific LoRA blocks would help me, just cropping the...

Furkan Gözükara SECourses•12/13/24, 8:16 PM

i tested it

Furkan Gözükara SECourses•12/13/24, 8:17 PM

it doesnt help

Furkan Gözükara SECourses•12/13/24, 8:17 PM

certain blocks

Furkan Gözükara SECourses•12/13/24, 8:17 PM

even single layer training still turns entire model into your face :d

Zhigarev•12/13/24, 10:29 PM

Could someone tell me if stylistic Loras are applied correctly to your fine-tuned models? Because on my own model, Loras don't work with the same settings that work on the shared flux.dev model. ... If it turns out that Loras don't work with fine-tuned models, that would be a failure...))

Yyolkypalky I have a Flux fine-tuned on a person, and I'm trying to pair with a LoRA trained...

Zhigarev•12/13/24, 10:32 PM

I understand this is a real problem with fine-tuned models?? Why was this not mentioned anywhere? Creating custom Loras and applying them to my characters was the main plan... Why did no one say a word about this? This is a mega failure...

Zhigarev•12/13/24, 10:32 PM

yolkypalky•12/13/24, 10:51 PM

@Zhigarev yeah this has been a real issue for me. The only thing that helps is disabling the last double block of the LoRA but it also really hurts the style

Yyolkypalky <@1237671795432226822> yeah this has been a real issue for me. The only thing th...

Zhigarev•12/13/24, 10:53 PM

Unfortunately, this destroys the entire pipeline I had planned for future potential clients, as I won’t even be able to apply something like a branded armored t-shirt from a specific company to them... let alone a vintage photography style, like in this example...

Zhigarev•12/13/24, 10:54 PM

What are the options for this mistake@Furkan Gözükara SECourses

ZZhigarev What are the options for this mistake<@205854764540362752>

Furkan Gözükara SECourses•12/13/24, 10:55 PM

loras overwriting your character

happened again: maybe turn off memory efficient save? Using memory efficient save file: /workspace/

Similar Threads

Similar Threads

Similar Threads