Software Engineering Courses (SECourses)•13mo ago

Thanks! It took around 15 minutes. I have started the training 🤞

Thanks! It took around 15 minutes. I have started the training

DD4UNTLESS How long does the massed compute "Initializing" take? It's been more than 10 min...

Furkan Gözükara SECourses•12/22/24, 10:07 AM

last time it took 13 for me

DD4UNTLESS Thanks! It took around 15 minutes. I have started the training 🤞

Furkan Gözükara SECourses•12/22/24, 10:08 AM

after init training takes me like 3 minutes to start including model download

D4UNTLESSOP•12/22/24, 11:06 AM

DD4UNTLESS Click to see attachment

D4UNTLESSOP•12/22/24, 11:07 AM

@Furkan Gözükara SECourses

DD4UNTLESS Click to see attachment

Furkan Gözükara SECourses•12/22/24, 11:15 AM

what config

Furkan Gözükara SECourses•12/22/24, 11:15 AM

did you generate samples?

Furkan Gözükara SECourses•12/22/24, 11:16 AM

are your all images 1024x1024?

D4UNTLESSOP•12/22/24, 11:21 AM

Batch 7 config. Dreambooth training. Images are in different resolutions. I turned on bucketing

DD4UNTLESS Batch 7 config. Dreambooth training. Images are in different resolutions. I turn...

Furkan Gözükara SECourses•12/22/24, 11:23 AM

that is why

Furkan Gözükara SECourses•12/22/24, 11:23 AM

reduce batch size to 6

Furkan Gözükara SECourses•12/22/24, 11:23 AM

batch size 7 barely fits when all 1024x1024 and no bucket

FFurkan Gözükara SECourses did you generate samples?

D4UNTLESSOP•12/22/24, 11:43 AM

You mentioned in the tutorial to not do sampling. I just followed your tutorial.

FFurkan Gözükara SECourses reduce batch size to 6

D4UNTLESSOP•12/22/24, 11:43 AM

Can I use batch 1 48gb on A6000? Will it take longer?

Timson•12/22/24, 12:07 PM

Btw, I still don't understand what's the point of batching if you have 100% CUDA cores utilisation on batch size 1?

DD4UNTLESS Can I use batch 1 48gb on A6000? Will it take longer?

Furkan Gözükara SECourses•12/22/24, 2:15 PM

yes you can use

Furkan Gözükara SECourses•12/22/24, 2:15 PM

it will take a little bit longer

TTimson Btw, I still don't understand what's the point of batching if you have 100% CUDA...

Furkan Gözükara SECourses•12/22/24, 2:15 PM

it still slightly improves speed

Furkan Gözükara SECourses•12/22/24, 2:16 PM

but for quality it reduces a little bit

Furkan Gözükara SECourses•12/22/24, 2:16 PM

you can calculate

Furkan Gözükara SECourses•12/22/24, 2:16 PM

batch size 1 = 6.4 second / it = 7*6.4 second for 7 samples

Furkan Gözükara SECourses•12/22/24, 2:16 PM

batch size 7 = 29.5 second / it

Furkan Gözükara SECourses•12/22/24, 2:16 PM

also i recommend now L40 with our coupon now only 49 cents

D4UNTLESSOP•12/22/24, 3:35 PM

I'm using cropping + padding to bring all my images to 1024x1024
So I'm not gonna use bucketing. Should I go with batch size 6?

DD4UNTLESS I'm using cropping + padding to bring all my images to 1024x1024 So I'm not gonn...

Furkan Gözükara SECourses•12/22/24, 4:16 PM

you shouldnt get

Furkan Gözükara SECourses•12/22/24, 4:16 PM

i tested many times

Furkan Gözükara SECourses•12/22/24, 4:16 PM

batch size 7 should work no bucket @ 1024 px

DD4UNTLESS Click to see attachment

D4UNTLESSOP•12/22/24, 4:36 PM

So this error was because of different resolutions? If I use all images with 1024x1024 now (no bucketing) I will not get errors at batch size 7? Just need to get confirmation from you

DD4UNTLESS So this error was because of different resolutions? If I use all images with 102...

Furkan Gözükara SECourses•12/22/24, 4:39 PM

yes

Furkan Gözükara SECourses•12/22/24, 4:39 PM

it was using more vram

FFurkan Gözükara SECourses yes

D4UNTLESSOP•12/22/24, 6:18 PM

ok thanks! I'll try again after a few hours

johnasdf.•12/22/24, 6:54 PM

Has anyone gotten this message when fine-tuning a Flux model? ValueError: mutable default <class 'timm.models.maxxvit.MaxxVitConvCfg'> for field conv_cfg is not allowed: use default_factory

Jjohnasdf.Has anyone gotten this message when fine-tuning a Flux model? ValueError: mutab...

Furkan Gözükara SECourses•12/22/24, 6:58 PM

which model ?

johnasdf.•12/22/24, 6:59 PM

Initially I fine-tuned with AGFlux_v21, a de-distilled model. Now I am trying to take a second pass on it.

johnasdf.•12/22/24, 7:02 PM

ChatGPT thinks it's either timm, Torch, or Kohya

johnasdf.•12/22/24, 7:04 PM

I noticed there are two venvs created when I ran the kohya setup v17... venv, and venv3. Is that what was intended to happen?

Jjohnasdf.Initially I fine-tuned with AGFlux_v21, a de-distilled model. Now I am trying to...

Furkan Gözükara SECourses•12/22/24, 7:06 PM

report to kohya

Furkan Gözükara SECourses•12/22/24, 7:06 PM

some models are not supported

johnasdf.•12/22/24, 7:07 PM

But I fine-tuned the model a few weeks ago, and then I just downloaded it from my huggingface account.

Jjohnasdf.But I fine-tuned the model a few weeks ago, and then I just downloaded it from m...

Furkan Gözükara SECourses•12/22/24, 7:07 PM

in that case

Furkan Gözükara SECourses•12/22/24, 7:08 PM

try model on swarmui

Furkan Gözükara SECourses•12/22/24, 7:08 PM

see if working

johnasdf.•12/22/24, 7:38 PM

Yeah. it works

johnasdf.•12/22/24, 7:52 PM

I got it to work. I just needed to upgrade timm to the latest version, which is 1.0.12. It had installed 0.6.12. If you want to update the Runpod installer.

johnasdf.•12/22/24, 7:54 PM

When you trained the de-distilled model did you use guidance > 1?

Jjohnasdf.When you trained the de-distilled model did you use guidance > 1?

Furkan Gözükara SECourses•12/22/24, 10:23 PM

yes

Furkan Gözükara SECourses•12/22/24, 10:23 PM

use 3.5 i think it was best

johnasdf.•12/22/24, 10:35 PM

Thank you

FFurkan Gözükara SECourses yes

D4UNTLESSOP•12/22/24, 11:53 PM

Thanks for the help! Last time I got an error at 32 steps.
I combined padding & cropping to bring all my images to 1014x1024. I used batch 7 config and last time I checked (1 hour ago), it was already at 250 steps

DD4UNTLESS Thanks for the help! Last time I got an error at 32 steps. I combined padding & ...

Furkan Gözükara SECourses•12/23/24, 12:35 AM

great

Furkan Gözükara SECourses•12/23/24, 12:35 AM

250 steps = 250 *7 samples

Thanks! It took around 15 minutes. I have started the training 🤞

Similar Threads

Similar Threads

Similar Threads