Software Engineering Courses (SECourses)•11mo ago

Hi! Don't know which channel to write this on. I'm training a lora and I'm wondering if I can stop t

Hi! Don't know which channel to write this on. I'm training a lora and I'm wondering if I can stop the training and start a training on the latest save like you can when training checkpoint?

EElli_Oaken Hi! Don't know which channel to write this on. I'm training a lora and I'm wonde...

Elli_OakenOP•2/20/25, 8:34 AM

Save training state (including optimizer states etc.) when saving models
Save training state (not enabled)

Save training state (including optimizer states etc.) on train end
Save training state at end of training (not enabled)

Resume from saved training state (path to "last-state" state folder)
Saved state to resume training from

MetaKan•2/20/25, 9:09 AM

I have finetuned flux-dev with 180 images of a character and 180 photos from various angles. Maybe it's not the best dataset because it didn’t have varied backgrounds. Almost all the images were just of the face without the body.
I'm having quite a few issues with prompts where I request a stylization. I trained a batch with 100 epochs and have 4 safetensors—I’ve tested them all, and I’m experiencing the same issues across the board.
When I ask for a stylized result, like a cartoon or anime style, it struggles to adhere to the prompt and often generates random things instead.
What could be happening? Thanks a lot!

MMetaKan I have finetuned flux-dev with 180 images of a character and 180 photos from var...

Timson•2/20/25, 9:28 AM

this is normal, nothing is wrong. You trained the model on realism data and it is inclined to generate it.

Timson•2/20/25, 9:29 AM

18000 steps is a lot, you may want to try earlier checkpoints

Timson•2/20/25, 9:31 AM

you may also try different prompt engineering, generate a lot of images and cherrypick from them secondary dataset to include in traning of second gen model with more stilyzed images in training data

TTimson 18000 steps is a lot, you may want to try earlier checkpoints

MetaKan•2/20/25, 10:36 AM

yes I did it

MetaKan•2/20/25, 10:36 AM

I did imagine that. But I watched @Furkan Gözükara SECourses tutorial and saw that he had some incredibly stylized images... I think he also only trained with realistic images. That's where my doubt comes from

EElli_Oaken Hi! Don't know which channel to write this on. I'm training a lora and I'm wonde...

Furkan Gözükara SECourses•2/20/25, 10:48 AM

yes buyt i never tried

Furkan Gözükara SECourses•2/20/25, 10:48 AM

its option is here

Furkan Gözükara SECourses•2/20/25, 10:49 AM

MMetaKan I have finetuned flux-dev with 180 images of a character and 180 photos from var...

Furkan Gözükara SECourses•2/20/25, 10:49 AM

yes flux is very realistic

Furkan Gözükara SECourses•2/20/25, 10:49 AM

therefore i recommend 2 step training for stylized images

Furkan Gözükara SECourses•2/20/25, 10:49 AM

first train, generate some good stylized images, and retrain on them

MMetaKan I did imagine that. But I watched <@205854764540362752> tutorial and saw that h...

Furkan Gözükara SECourses•2/20/25, 10:50 AM

i had 256 images and dataset was variety . also still i had hard time and needed to find prompts

MetaKan•2/20/25, 10:52 AM

thank you sir!

Furkan Gözükara SECourses•2/20/25, 10:56 AM

The grid test I am analyzing to find newer FLUX DEV training. This is not even including the first 10 different trainings i made

1176 images. I think we gonna have a better workflow than what we have with using more VRAM and more training time.

Renzo•2/21/25, 1:46 PM

In case someone is interested in creating LoRAs from Lumina2: https://huggingface.co/sayakpaul/trained-lumina2-lora-yarn, https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_lumina2.md (the code).

GitHub

diffusers/examples/dreambooth/README_lumina2.md at main · huggingfa...

Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch and FLAX. - huggingface/diffusers

FFurkan Gözükara SECourses The grid test I am analyzing to find newer FLUX DEV training. This is not even i...

Mathew•2/21/25, 6:49 PM

Its Lora train or checkpoint ?

MMathew Its Lora train or checkpoint ?

Furkan Gözükara SECourses•2/21/25, 11:46 PM

fine tuning atm

Furkan Gözükara SECourses•2/21/25, 11:46 PM

after that i plan lora

ghstoic•2/22/25, 10:42 AM

For multi-GPU training, does the set-up need 80gb vram TOTAL, or does EACH gpu need 80gb vram?

Gghstoic For multi-GPU training, does the set-up need 80gb vram TOTAL, or does EACH gpu n...

Furkan Gözükara SECourses•2/22/25, 11:23 AM

for fine tuning

Furkan Gözükara SECourses•2/22/25, 11:23 AM

each gpu 80 gb

Furkan Gözükara SECourses•2/22/25, 11:23 AM

for lora not necessary 48 gb gpus works great

FFurkan Gözükara SECourses each gpu 80 gb

ghstoic•2/22/25, 12:25 PM

Gottcha, thank you!

ghstoic•2/22/25, 12:27 PM

Out of interest, what causes the VRAM per GPU requirements to jump up so much? Just curious on what's going on in the background processes that requires VRAM requirement to go from 24gb VRAM per GPU from single training to 80gb VRAM per GPU for muli-traning

Gghstoic Out of interest, what causes the VRAM per GPU requirements to jump up so much? J...

Furkan Gözükara SECourses•2/22/25, 12:45 PM

extremely good question

Furkan Gözükara SECourses•2/22/25, 12:45 PM

i asked this to kohya

Furkan Gözükara SECourses•2/22/25, 12:46 PM

he was also not sure

Timson•2/22/25, 11:51 PM

How does multy GPU trainign work anyways?
Each GPU has a copy of a trained model and perform traning steps independantly, so how in the end we are not gettign two separate models?

Timson•2/22/25, 11:52 PM

they cannot possibly synchronize model state after each step, that woudl take ages (minutes)

Timson•2/22/25, 11:55 PM

And if the models are just merged as last step - would that mean the final model can have questionable quality (missed local minimas and such)?

TTimson How does multy GPU trainign work anyways? Each GPU has a copy of a trained model...

Furkan Gözükara SECourses•2/23/25, 10:01 AM

with diffuser models

Furkan Gözükara SECourses•2/23/25, 10:01 AM

each models loaded fully into vram

Furkan Gözükara SECourses•2/23/25, 10:02 AM

like flux sd etc

Furkan Gözükara SECourses•2/23/25, 10:02 AM

so we have exact replicate

FFurkan Gözükara SECourses each models loaded fully into vram

Timson•2/23/25, 10:41 AM

yeah, but how the state of each model loaded into GPU sync up?

Timson•2/23/25, 10:41 AM

So the training result in single model, not multiple models

TTimson yeah, but how the state of each model loaded into GPU sync up?

Furkan Gözükara SECourses•2/23/25, 10:45 AM

they are snyched at steps

Furkan Gözükara SECourses•2/23/25, 10:45 AM

so it is being increased batch size

novsky•2/23/25, 4:31 PM

long time no see. I had trained few flux loras earlier, and had to reinstall my windows recently. After that I had to reinstall kohya, and now can't train anymore. First it didn't create venv (idk, maybe it's because latest installer or something else). I've installed kohya from original repo, and it started to work. But now I have another problem. Training speed reduced. Much. ~500s/it. I guess it's because it's using ram or something. Earlier I noticed it's caching t5 and clip-l to ram (to free vram), now there is no such notice. Any suggestions?

novsky•2/23/25, 4:32 PM

gpu 4060ti 16gb, ram 64 gb

novsky•2/23/25, 4:32 PM