"clip_skip": 1,

MMaxivi "clip_skip": 1,

F

Furkan Gözükara SECourses•9/1/24, 12:44 AM

ye clip skip there

F

Furkan Gözükara SECourses•9/1/24, 12:44 AM

i mean about removed params

M

MaxiviOP•9/1/24, 12:45 AM

you mean when it goes into training the toml?

MMaxivi you mean when it goes into training the toml?

F

Furkan Gözükara SECourses•9/1/24, 12:45 AM

in saved json file

F

Furkan Gözükara SECourses•9/1/24, 12:45 AM

i made a fresh one

M

MaxiviOP•9/1/24, 12:45 AM

whatever was in the json gets trained on the previous kohya one, the latest it doesnt

M

MaxiviOP•9/1/24, 12:45 AM

lemme check patreon

F

Furkan Gözükara SECourses•9/1/24, 12:46 AM

i am doing new tests

F

Furkan Gözükara SECourses•9/1/24, 12:46 AM

clicp skip discerte value

M

MaxiviOP•9/1/24, 12:46 AM

which post did you update it on?

MMaxivi which post did you update it on?

F

Furkan Gözükara SECourses•9/1/24, 12:47 AM

i didnt update again v21 is latest

F

Furkan Gözükara SECourses•9/1/24, 12:47 AM

i will update tomorrow hopefully

M

MaxiviOP•9/1/24, 12:47 AM

oh no worries i will check tomorrow

F

Furkan Gözükara SECourses•9/1/24, 12:48 AM

i will test clip skip 2 as well

M

MaxiviOP•9/1/24, 12:52 AM

also xformers seems to be worksing

M

MaxiviOP•9/1/24, 12:52 AM

working*

MMaxivi working*

F

Furkan Gözükara SECourses•9/1/24, 12:53 AM

i didnt see its benefit in terms of speed or vram so didnt bother

M

MaxiviOP•9/1/24, 12:53 AM

seems faster for me than sdpa

M

MaxiviOP•9/1/24, 12:53 AM

and 1gb less vram

F

Furkan Gözükara SECourses•9/1/24, 12:53 AM

i should test on windows

F

Furkan Gözükara SECourses•9/1/24, 12:53 AM

i tested on ubuntu :d

F

Furkan Gözükara SECourses•9/1/24, 12:54 AM

quality also matters

M

MaxiviOP•9/1/24, 12:54 AM

ohhh maybe windows with the new torch

M

MaxiviOP•9/1/24, 12:54 AM

testing quality now

M

MaxiviOP•9/1/24, 12:54 AM

first try on xformers

F

Furkan Gözükara SECourses•9/1/24, 12:55 AM

which xformers you installed

M

MaxiviOP•9/1/24, 12:56 AM

let me check for you

F

Furkan Gözükara SECourses•9/1/24, 12:56 AM

i got xformers==0.0.27.post2+cu118

F

Furkan Gözükara SECourses•9/1/24, 12:56 AM

gonna train on this

M

MaxiviOP•9/1/24, 12:59 AM

just checked

M

MaxiviOP•9/1/24, 12:59 AM

yeah using same one

M

MaxiviOP•9/1/24, 12:59 AM

pip install xformers==0.0.27.post2

F

Furkan Gözükara SECourses•9/1/24, 1:01 AM

ye uses lesser vram and looks alitt bit fater

F

Furkan Gözükara SECourses•9/1/24, 1:01 AM

if same quality i will update all configs :d

F

Furkan Gözükara SECourses•9/1/24, 1:02 AM

ok vram usage became same :d

F

Furkan Gözükara SECourses•9/1/24, 1:03 AM

M

MaxiviOP•9/1/24, 1:05 AM

i think for linux xformers works fine because of the torch version

M

MaxiviOP•9/1/24, 1:06 AM

but for windows i think its using none, no xformers or sdpa ?

M

MaxiviOP•9/1/24, 1:06 AM

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

M

MaxiviOP•9/1/24, 1:06 AM

i get this here on widnows

M

MaxiviOP•9/1/24, 1:08 AM

but its much faster for me than sdpa

M

MaxiviOP•9/1/24, 1:08 AM

interesting

FFurkan Gözükara SECourses depends on how many epochs

G

GomezBro•9/1/24, 1:39 AM

I am unsure how epoch works. I see your settings always has 200, why change it? Is 1000 images too much to train a style?

Why touch or change epochs? Thats how steps are determined?

G

GomezBro•9/1/24, 2:42 AM

Can I get that coupon and website please? @Dr. Furkan Gözükara

GGomezBro I am unsure how epoch works. I see your settings always has 200, why change it?...

D

druhl•9/1/24, 5:21 AM

From what I understand (correct me, if I am wrong, Doc), an epoch equals how many times an image in the input dataset is trained.
So suppose you have an input dataset of 15 images, and you select an epoch of 200, each image will be trained 200 times, so the No. of steps = 15x200 = 3000 (3000 steps is a trade-off figure b/w time and quality that the Doc likes to use in his videos).

Now, the time taken for each step will depend upon your GPU. So, say a 12GB GPU trains with Flux at 12 second/ iteration. Then, the total time =
3000 x 12 = 36,000 secs (approx. 10hrs.)

D

druhl•9/1/24, 5:23 AM

The above is when you use repeat = 1,
So no. of steps = epoch x repeats x no. of images in input dataset

D

druhl•9/1/24, 5:25 AM

And time taken (in hrs.) = (no. of steps x time taken for each iteration by the GPU)/3600

D

druhl•9/1/24, 5:28 AM

What I do not understand (yet!), however, is:
1) what difference it makes if you use 3000 steps versus, say 7000 steps (increase the epoch for the same no. of input images. My assumption is it takes more time, but the quality will be better to a certain extent, before it gives diminishing returns and then peaks).
2) Use less images & more epoch, versus more images and less epoch (However, the no. of steps is maintained the same. Eg. 15x200=3000, and 50x60=also3000. Here, I make no assumption and would like to hear from Doc).

D

dxqb•9/1/24, 6:45 AM

1) largely correct, but there is no peak, there is a cliff. you can overfit diffusion models quite easily

D

dxqb•9/1/24, 6:50 AM

2) there is no clear answer, it's an art. Less images tend to give you an inflexible result, more images tend to be harder to train until the model picks up the details. You often need more steps in total for many images than for a few, meaning not reduce the epochs as much as you have increased the number of images. Larger batch sizes can help with (many) more images. Adjustments in LR might be necessary.

Similar Threads

Similar Threads

Similar Threads