Software Engineering Courses (SECourses)•13mo ago

Is it normal that saving flux dev fp16 checkpoint takes 6 minutes? I've tested the drive on that rem

Is it normal that saving flux dev fp16 checkpoint takes 6 minutes?
I've tested the drive on that remote to have above 500MB/s writing speed, I don't see any CPU or GPU load during this operation, what could be the bottleneck?

TimsonOP•12/14/24, 6:14 PM

what is the "normal" duration of this operation?

Latentnaut•12/14/24, 6:35 PM

Hello group, do you think a finetune training of this product will capture all these little letters and details?

TTimson Is it normal that saving flux dev fp16 checkpoint takes 6 minutes? I've tested t...

TimsonOP•12/14/24, 6:45 PM

figured this out, I cheeped out on 16GB RAM + 16 GB swap that gets 99% filled during checkpoint save. Lesson learned, seems you need 32GB RAM for seamless training :heher:

Nneo do we got any alternatives like Massed Compute? other than runpod

Renzo•12/14/24, 7:38 PM

I have used https://www.hyperstack.cloud/. They have reasonable prices. But you have to configure everything mostly from scratch.

TTimson but there is no fundamental problem on training with mixed images sizes in a set...

Furkan Gözükara SECourses•12/14/24, 11:31 PM

there shouldnt be

Ccyberbol What is it new option ReActor ?

Furkan Gözükara SECourses•12/14/24, 11:31 PM

face swap

TTimson Is it normal that saving flux dev fp16 checkpoint takes 6 minutes? I've tested t...

Furkan Gözükara SECourses•12/14/24, 11:32 PM

no savings way faster for me

Furkan Gözükara SECourses•12/14/24, 11:32 PM

like few seconds

LLatentnaut Hello group, do you think a finetune training of this product will capture all t...

Furkan Gözükara SECourses•12/14/24, 11:32 PM

not likely

Furkan Gözükara SECourses•12/14/24, 11:32 PM

some text really small

TTimson figured this out, I cheeped out on 16GB RAM + 16 GB swap that gets 99% filled du...

Furkan Gözükara SECourses•12/14/24, 11:32 PM

actually get 64 gb

Furkan Gözükara SECourses•12/15/24, 12:09 AM

Furkan Gözükara SECourses•12/15/24, 1:12 AM

published : https://www.patreon.com/posts/117990364

Patreon

MMAudio 1-Click Installers for Windows, RunPod and Massed Compute -...

Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon

Furkan Gözükara SECourses•12/15/24, 1:12 AM

Sipriyani•12/15/24, 1:18 AM

and interesting method for , consistent images and reposing ... however, each scene will need a lora training. https://www.youtube.com/watch?v=MbQv8zoNEfY

Furkan Gözükara SECourses•12/15/24, 1:32 AM

best consistent way is training

Furkan Gözükara SECourses•12/15/24, 1:32 AM

you can do multi step to get what you want

QM•12/15/24, 1:33 AM

I have a 6x 4090 setup on Win 11 - is there a way to use all GPUs for finetune training ?

QQM I have a 6x 4090 setup on Win 11 - is there a way to use all GPUs for finetune t...

Furkan Gözükara SECourses•12/15/24, 1:34 AM

nope

Furkan Gözükara SECourses•12/15/24, 1:34 AM

fine tuning need 80 gb gpus

Furkan Gözükara SECourses•12/15/24, 1:34 AM

but you can do 6 parallel training

Furkan Gözükara SECourses•12/15/24, 1:35 AM

i got to sleep now hopefully see you tomorrow

QM•12/15/24, 1:35 AM

is it possible to build up a finetuned or LoRA model - i.e. pictures of me, wife kids etc?

QQM is it possible to build up a finetuned or LoRA model - i.e. pictures of me, wife...

Furkan Gözükara SECourses•12/15/24, 1:38 AM

yes

Furkan Gözükara SECourses•12/15/24, 1:38 AM

have all of you in same pictures as well during training

Furkan Gözükara SECourses•12/15/24, 1:38 AM

and caption them

QM•12/15/24, 1:38 AM

have a good night

Furkan Gözükara SECourses•12/15/24, 1:40 AM

like ohwx man bbuk woman qwxx kid

Furkan Gözükara SECourses•12/15/24, 1:40 AM

i didnt test this but logic is like this

Furkan Gözükara SECourses•12/15/24, 1:40 AM

FFurkan Gözükara SECourses like ohwx man bbuk woman qwxx kid

QM•12/15/24, 1:40 AM

thank you

FFurkan Gözükara SECourses but you can do 6 parallel training 😄

QM•12/15/24, 1:54 AM

I am curious to maximise the PCs ability - with 6x 4090 and 512 DDR5 RAM (PCIe 5) - to train models faster

FFurkan Gözükara SECourses you can do multi step to get what you want 🙂

Sipriyani•12/15/24, 1:57 AM

doctor what exactly you mean with multi step?

Sipriyani•12/15/24, 1:59 AM

also another question, can I re-finetune, a flux model file which is already fine tuned?

dsienra•12/15/24, 3:54 AM

https://civitai.com/models/1034630/hunyuan-video-arnold-schwarzenegger-lora?modelVersionId=1160430 Great hunyuan lora trained only with 15 images, this video model generalize very well. very promising

QQM I am curious to maximise the PCs ability - with 6x 4090 and 512 DDR5 RAM (PCIe 5...

XCogni•12/15/24, 10:16 AM

sell the 6 4090s, buy 2 6000adas, or wait for the new blackwell chips coming out

FFurkan Gözükara SECourses like few seconds

TimsonOP•12/15/24, 11:24 AM

and what is you time for swapping fine-tuned full checkpoints in SwarmUI? Mine like 120s per switch

There is this lines in debug log during model switch are they ok? clip_l is loaded and working during generation, and it says something about casting although I'm saving checkpoints as fp16

FFurkan Gözükara SECourses published : https://www.patreon.com/posts/117990364

J2_Nedo•12/15/24, 12:14 PM

thanx this is really good for our own made videos

QQM I am curious to maximise the PCs ability - with 6x 4090 and 512 DDR5 RAM (PCIe 5...

Furkan Gözükara SECourses•12/15/24, 2:27 PM

maybe multi gpu lora but i am not much hopeful

SSipriyani doctor what exactly you mean with multi step?

Furkan Gözükara SECourses•12/15/24, 2:28 PM

like train 1 time generate pictures pick good ones train another time

SSipriyani also another question, can I re-finetune, a flux model file which is already fin...

Furkan Gözükara SECourses•12/15/24, 2:28 PM

i wouldnt recommend

Furkan Gözükara SECourses•12/15/24, 2:28 PM

flux already very prone to overfit

Renzo•12/15/24, 2:51 PM

@Dr. Furkan Gözükara, in which cases do you use YOLOv9? I have observed that you add it to some prompts and I wonder what makes you decide to add it.

RRenzo @Dr. Furkan Gözükara, in which cases do you use YOLOv9? I have observed that you...

TimsonOP•12/15/24, 2:53 PM

the cases where character face is likely to be small in the image composition

TimsonOP•12/15/24, 2:57 PM

I'm experimenting with fine-tuned models and so far I'm not entirly convinced they are better than loras in all cases, especially without additional inpainting techniques.
Maybe they are still under-trained, but I tested 1800, 2000, 2500 training steps on different training images set sizes (17, 25, 40) with mixed results, some seeds yield good likeness but some miss completely with the same prompts.