Software Engineering Courses (SECourses)•15mo ago

I've got a feeling I'm going to run out of space on massed compute lol. Can I move the completed che

I've got a feeling I'm going to run out of space on massed compute lol. Can I move the completed checkpoints to drive and delete them while the training is ongoing?

Furkan Gözükara SECourses•10/12/24, 2:06 PM

Ddruhl I've got a feeling I'm going to run out of space on massed compute lol. Can I mo...

Furkan Gözükara SECourses•10/12/24, 2:07 PM

sure

FFurkan Gözükara SECourses Click to see attachment

druhlOP•10/12/24, 2:09 PM

How things started

Screenshot_2024-10-12-19-38-37-13_f9ee0578fe1cc94de7482bd41accb329.jpg

Furkan Gözükara SECourses•10/12/24, 2:09 PM

noting the best checkpoints

Ddruhl How things started 😄

Furkan Gözükara SECourses•10/12/24, 2:09 PM

ye when i look bad the resemblance was too bad :d

druhlOP•10/12/24, 2:10 PM

Flux has spoilt us all lol

Furkan Gözükara SECourses•10/12/24, 2:10 PM

true

Furkan Gözükara SECourses•10/12/24, 2:10 PM

now sd 3.1 has to be good

FFurkan Gözükara SECourses noting the best checkpoints

druhlOP•10/12/24, 2:11 PM

How many epochs did you train for 256 image BS7 (fine-tune)?

druhlOP•10/12/24, 2:12 PM

I have 225 imgs & I'm now saving every 2 epoch 'cuz it overtrains super fast

Ddruhl How many epochs did you train for 256 image BS7 (fine-tune)?

Furkan Gözükara SECourses•10/12/24, 2:18 PM

best one is 50 epoch

Ddruhl I have 225 imgs & I'm now saving every 2 epoch 'cuz it overtrains super fast 😀

Furkan Gözükara SECourses•10/12/24, 2:18 PM

it sounds your LR too high

Furkan Gözükara SECourses•10/12/24, 2:18 PM

or images are too duplicate :d

FFurkan Gözükara SECourses it sounds your LR too high

druhlOP•10/12/24, 2:22 PM

Using 4e-6, or 0.000004, which was recommended. But hey, 50*256 would be 12,800 steps! Do you mean BS7 would mean 12,800/7 ~ 1,828 steps approx?

Ddruhl Using 4e-6, or 0.000004, which was recommended. But hey, 50*256 would be 12,800 ...

Furkan Gözükara SECourses•10/12/24, 2:26 PM

yep

Furkan Gözükara SECourses•10/12/24, 2:26 PM

i will publish my grids generating them

Furkan Gözükara SECourses•10/12/24, 2:26 PM

also my findings

druhlOP•10/12/24, 3:39 PM

Now tuning two checkpoints side by side, one on flux_dev, another on dev_de_distill.

druhlOP•10/12/24, 4:50 PM

The de-distill trains a tad bit faster it seems.

druhlOP•10/12/24, 4:50 PM

This is what I made earlier with Flux Pro 1.1

druhlOP•10/12/24, 4:51 PM

Same prompt for Flux_dev:

druhlOP•10/12/24, 4:51 PM

druhlOP•10/12/24, 4:52 PM

Wish we had access to Pro as well.

dsienra•10/12/24, 5:20 PM

I'm training a lora tree people 2 same class 1 different class with regularization on flux-dev-de-distill, lets see if regularization images works with flux-dev-de-distill

dsienra•10/12/24, 5:21 PM

the regularization are person class

dsienra•10/12/24, 5:22 PM

the bleeding between subjects was fixed with flux-dev-de-distill but idk yet if works with regularization

dsienra•10/12/24, 5:36 PM

@Dr. Furkan Gözükara flux-dev-de-distill what I saw was that at 30 epoch's the resemblances was reduced with regularization images, lets see if a full 160 epochs fixes that.

Ddsienra @Dr. Furkan Gözükara flux-dev-de-distill what I saw was that at 30 epoch's the r...

druhlOP•10/12/24, 5:47 PM

How many images are there in your dataset?

Ddsienra I'm training a lora tree people 2 same class 1 different class with regularizati...

druhlOP•10/12/24, 5:48 PM

What tool do you use to make this curve? I don't let logs save at all lol. Perhaps I should reconsider.

dsienra•10/12/24, 5:52 PM

just tensorboard on kohya, 10 or 11 pictures per subject, I have a folder with real people as regularization, 2000 images different ages, genders and races

dsienra•10/12/24, 5:53 PM

I downloaded the regularization dataset a time ago with captions

Ddsienra just tensorboard on kohya, 10 or 11 pictures per subject, I have a folder with r...

druhlOP•10/12/24, 5:53 PM

Awesome!

druhlOP•10/12/24, 5:56 PM

For me, that option says start even though the training is on

@dsienra

dsienra•10/12/24, 5:58 PM

just press it

dsienra•10/12/24, 5:58 PM

it will start tensorboard

dsienra•10/12/24, 5:58 PM

is independent of trainig

dsienra•10/12/24, 6:00 PM

dsienra•10/12/24, 6:01 PM

just select the current training in the bottom or enable all to compare the loss curve with past trainings

dsienra•10/12/24, 6:01 PM

is just a guide

dsienra•10/12/24, 6:07 PM

in theory it should go down until starts overtraining and in that case it should go up again, when the training just starts the loos goes up and down, that is normal on the first epochs, but overall the graphic should go down during the entire training

deepblue_1337•10/12/24, 7:54 PM

Hello,

I am trying to follow the Patreon post on finetuning.
https://www.patreon.com/posts/huge-news-for-as-113531833

Patreon

Huge news for Kohya GUI - Now you can fully Fine Tune / DreamBooth ...

Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon

deepblue_1337•10/12/24, 7:55 PM

it says same process as lora, only different config. Do I still use the LoRa tab in the webinterface or do I use dreambooth?

Thanks
Daniel

Ddeepblue_1337 it says same process as lora, only different config. Do I still use the LoRa tab...

druhlOP•10/12/24, 7:59 PM

Do dreambooth

deepblue_1337•10/12/24, 8:01 PM

I have a RTX 3060 and am getting ~45sec/it, which would take quite long for 9200 steps, is that normal?

deepblue_1337•10/12/24, 8:02 PM

nvitop is showing >90% GPU utilization

Ddsienra @Dr. Furkan Gözükara flux-dev-de-distill what I saw was that at 30 epoch's the r...

Furkan Gözükara SECourses•10/12/24, 9:56 PM

nice test

Ddeepblue_1337 it says same process as lora, only different config. Do I still use the LoRa tab...

Furkan Gözükara SECourses•10/12/24, 9:57 PM

dreambooth tab

Ddeepblue_1337 I have a RTX 3060 and am getting ~45sec/it, which would take quite long for 9200...

Furkan Gözükara SECourses•10/12/24, 9:57 PM

yes normal

Furkan Gözükara SECourses•10/12/24, 9:57 PM

rtx 3060 takes around 40 second / it in long run

FFurkan Gözükara SECourses nice test

dsienra•10/12/24, 10:04 PM

The training was finished, resemblance is almost perfect, but it still looks a little undertrained, with regularization images it needs much more steps or increase the learning rate to get resemblance. I will resume the training from 160 repetitions per image and I will report back, there was no bleeding between the trained subjects

I've got a feeling I'm going to run out of space on massed compute lol. Can I move the completed che

Similar Threads

Similar Threads

Similar Threads