Software Engineering Courses (SECourses)•13mo ago

How long does the massed compute "Initializing" take? It's been more than 10 minutes now

Timson•12/22/24, 9:32 AM

the bigger initialisation image - the longer it takes

10 minutes is sensible for larger desktop images

TTimson the bigger initialisation image - the longer it takes 🤷‍♂️ 10 minutes is sensib...

D4UNTLESSOP•12/22/24, 9:59 AM

Thanks! It took around 15 minutes. I have started the training

DD4UNTLESS How long does the massed compute "Initializing" take? It's been more than 10 min...

Furkan Gözükara SECourses•12/22/24, 10:07 AM

last time it took 13 for me

DD4UNTLESS Thanks! It took around 15 minutes. I have started the training 🤞

Furkan Gözükara SECourses•12/22/24, 10:08 AM

after init training takes me like 3 minutes to start including model download

D4UNTLESSOP•12/22/24, 11:06 AM

DD4UNTLESS Click to see attachment

D4UNTLESSOP•12/22/24, 11:07 AM

@Furkan Gözükara SECourses

DD4UNTLESS Click to see attachment

Furkan Gözükara SECourses•12/22/24, 11:15 AM

what config

Furkan Gözükara SECourses•12/22/24, 11:15 AM

did you generate samples?

Furkan Gözükara SECourses•12/22/24, 11:16 AM

are your all images 1024x1024?

D4UNTLESSOP•12/22/24, 11:21 AM

Batch 7 config. Dreambooth training. Images are in different resolutions. I turned on bucketing

DD4UNTLESS Batch 7 config. Dreambooth training. Images are in different resolutions. I turn...

Furkan Gözükara SECourses•12/22/24, 11:23 AM

that is why

Furkan Gözükara SECourses•12/22/24, 11:23 AM

reduce batch size to 6

Furkan Gözükara SECourses•12/22/24, 11:23 AM

batch size 7 barely fits when all 1024x1024 and no bucket

FFurkan Gözükara SECourses did you generate samples?

D4UNTLESSOP•12/22/24, 11:43 AM

You mentioned in the tutorial to not do sampling. I just followed your tutorial.

FFurkan Gözükara SECourses reduce batch size to 6

D4UNTLESSOP•12/22/24, 11:43 AM

Can I use batch 1 48gb on A6000? Will it take longer?

Timson•12/22/24, 12:07 PM

Btw, I still don't understand what's the point of batching if you have 100% CUDA cores utilisation on batch size 1?

DD4UNTLESS Can I use batch 1 48gb on A6000? Will it take longer?

Furkan Gözükara SECourses•12/22/24, 2:15 PM

yes you can use

Furkan Gözükara SECourses•12/22/24, 2:15 PM

it will take a little bit longer

TTimson Btw, I still don't understand what's the point of batching if you have 100% CUDA...

Furkan Gözükara SECourses•12/22/24, 2:15 PM

it still slightly improves speed

Furkan Gözükara SECourses•12/22/24, 2:16 PM

but for quality it reduces a little bit

Furkan Gözükara SECourses•12/22/24, 2:16 PM

you can calculate

Furkan Gözükara SECourses•12/22/24, 2:16 PM

batch size 1 = 6.4 second / it = 7*6.4 second for 7 samples

Furkan Gözükara SECourses•12/22/24, 2:16 PM

batch size 7 = 29.5 second / it

Furkan Gözükara SECourses•12/22/24, 2:16 PM

also i recommend now L40 with our coupon now only 49 cents

D4UNTLESSOP•12/22/24, 3:35 PM

I'm using cropping + padding to bring all my images to 1024x1024
So I'm not gonna use bucketing. Should I go with batch size 6?

DD4UNTLESS I'm using cropping + padding to bring all my images to 1024x1024 So I'm not gonn...

Furkan Gözükara SECourses•12/22/24, 4:16 PM

you shouldnt get

Furkan Gözükara SECourses•12/22/24, 4:16 PM

i tested many times

Furkan Gözükara SECourses•12/22/24, 4:16 PM

batch size 7 should work no bucket @ 1024 px

DD4UNTLESS Click to see attachment

D4UNTLESSOP•12/22/24, 4:36 PM

So this error was because of different resolutions? If I use all images with 1024x1024 now (no bucketing) I will not get errors at batch size 7? Just need to get confirmation from you

DD4UNTLESS So this error was because of different resolutions? If I use all images with 102...

Furkan Gözükara SECourses•12/22/24, 4:39 PM

yes

Furkan Gözükara SECourses•12/22/24, 4:39 PM

it was using more vram

FFurkan Gözükara SECourses yes

D4UNTLESSOP•12/22/24, 6:18 PM

ok thanks! I'll try again after a few hours

johnasdf.•12/22/24, 6:54 PM

Has anyone gotten this message when fine-tuning a Flux model? ValueError: mutable default <class 'timm.models.maxxvit.MaxxVitConvCfg'> for field conv_cfg is not allowed: use default_factory

Jjohnasdf.Has anyone gotten this message when fine-tuning a Flux model? ValueError: mutab...

Furkan Gözükara SECourses•12/22/24, 6:58 PM

which model ?

johnasdf.•12/22/24, 6:59 PM

Initially I fine-tuned with AGFlux_v21, a de-distilled model. Now I am trying to take a second pass on it.

johnasdf.•12/22/24, 7:02 PM

ChatGPT thinks it's either timm, Torch, or Kohya

johnasdf.•12/22/24, 7:04 PM

I noticed there are two venvs created when I ran the kohya setup v17... venv, and venv3. Is that what was intended to happen?

Jjohnasdf.Initially I fine-tuned with AGFlux_v21, a de-distilled model. Now I am trying to...

Furkan Gözükara SECourses•12/22/24, 7:06 PM

report to kohya

Furkan Gözükara SECourses•12/22/24, 7:06 PM

some models are not supported

johnasdf.•12/22/24, 7:07 PM

But I fine-tuned the model a few weeks ago, and then I just downloaded it from my huggingface account.

Jjohnasdf.But I fine-tuned the model a few weeks ago, and then I just downloaded it from m...

Furkan Gözükara SECourses•12/22/24, 7:07 PM

in that case

Furkan Gözükara SECourses•12/22/24, 7:08 PM

try model on swarmui

Furkan Gözükara SECourses•12/22/24, 7:08 PM

see if working

johnasdf.•12/22/24, 7:38 PM

Yeah. it works

johnasdf.•12/22/24, 7:52 PM

I got it to work. I just needed to upgrade timm to the latest version, which is 1.0.12. It had installed 0.6.12. If you want to update the Runpod installer.

johnasdf.•12/22/24, 7:54 PM

When you trained the de-distilled model did you use guidance > 1?

Jjohnasdf.When you trained the de-distilled model did you use guidance > 1?

Furkan Gözükara SECourses•12/22/24, 10:23 PM

yes

Furkan Gözükara SECourses•12/22/24, 10:23 PM

use 3.5 i think it was best

johnasdf.•12/22/24, 10:35 PM

Thank you

FFurkan Gözükara SECourses yes

D4UNTLESSOP•12/22/24, 11:53 PM

Thanks for the help! Last time I got an error at 32 steps.
I combined padding & cropping to bring all my images to 1014x1024. I used batch 7 config and last time I checked (1 hour ago), it was already at 250 steps

How long does the massed compute "Initializing" take? It's been more than 10 minutes now

Similar Threads

Similar Threads

Similar Threads