working*

MMaxivi working*

F

Furkan Gözükara SECourses•9/1/24, 12:53 AM

i didnt see its benefit in terms of speed or vram so didnt bother

M

MaxiviOP•9/1/24, 12:53 AM

seems faster for me than sdpa

M

MaxiviOP•9/1/24, 12:53 AM

and 1gb less vram

F

Furkan Gözükara SECourses•9/1/24, 12:53 AM

i should test on windows

F

Furkan Gözükara SECourses•9/1/24, 12:53 AM

i tested on ubuntu :d

F

Furkan Gözükara SECourses•9/1/24, 12:54 AM

quality also matters

M

MaxiviOP•9/1/24, 12:54 AM

ohhh maybe windows with the new torch

M

MaxiviOP•9/1/24, 12:54 AM

testing quality now

M

MaxiviOP•9/1/24, 12:54 AM

first try on xformers

F

Furkan Gözükara SECourses•9/1/24, 12:55 AM

which xformers you installed

M

MaxiviOP•9/1/24, 12:56 AM

let me check for you

F

Furkan Gözükara SECourses•9/1/24, 12:56 AM

i got xformers==0.0.27.post2+cu118

F

Furkan Gözükara SECourses•9/1/24, 12:56 AM

gonna train on this

M

MaxiviOP•9/1/24, 12:59 AM

just checked

M

MaxiviOP•9/1/24, 12:59 AM

yeah using same one

M

MaxiviOP•9/1/24, 12:59 AM

pip install xformers==0.0.27.post2

F

Furkan Gözükara SECourses•9/1/24, 1:01 AM

ye uses lesser vram and looks alitt bit fater

F

Furkan Gözükara SECourses•9/1/24, 1:01 AM

if same quality i will update all configs :d

F

Furkan Gözükara SECourses•9/1/24, 1:02 AM

ok vram usage became same :d

F

Furkan Gözükara SECourses•9/1/24, 1:03 AM

M

MaxiviOP•9/1/24, 1:05 AM

i think for linux xformers works fine because of the torch version

M

MaxiviOP•9/1/24, 1:06 AM

but for windows i think its using none, no xformers or sdpa ?

M

MaxiviOP•9/1/24, 1:06 AM

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

2024-08-31 19:02:48 WARNING  WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was    _cpp_lib.py:148
                             built for:
                                 PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.5.0.dev20240828+cu124)
                                 Python  3.10.11 (you have 3.10.11)
                               Please reinstall xformers (see
                             https://github.com/facebookresearch/xformers#installing-xformers)
                               Memory-efficient attention, SwiGLU, sparse and more won't be available.
                               Set XFORMERS_MORE_DETAILS=1 for more details

M

MaxiviOP•9/1/24, 1:06 AM

i get this here on widnows

M

MaxiviOP•9/1/24, 1:08 AM

but its much faster for me than sdpa

M

MaxiviOP•9/1/24, 1:08 AM

interesting

FFurkan Gözükara SECourses depends on how many epochs

G

GomezBro•9/1/24, 1:39 AM

I am unsure how epoch works. I see your settings always has 200, why change it? Is 1000 images too much to train a style?

Why touch or change epochs? Thats how steps are determined?

G

GomezBro•9/1/24, 2:42 AM

Can I get that coupon and website please? @Dr. Furkan Gözükara

GGomezBro I am unsure how epoch works. I see your settings always has 200, why change it?...

D

druhl•9/1/24, 5:21 AM

From what I understand (correct me, if I am wrong, Doc), an epoch equals how many times an image in the input dataset is trained.
So suppose you have an input dataset of 15 images, and you select an epoch of 200, each image will be trained 200 times, so the No. of steps = 15x200 = 3000 (3000 steps is a trade-off figure b/w time and quality that the Doc likes to use in his videos).

Now, the time taken for each step will depend upon your GPU. So, say a 12GB GPU trains with Flux at 12 second/ iteration. Then, the total time =
3000 x 12 = 36,000 secs (approx. 10hrs.)

D

druhl•9/1/24, 5:23 AM

The above is when you use repeat = 1,
So no. of steps = epoch x repeats x no. of images in input dataset

D

druhl•9/1/24, 5:25 AM

And time taken (in hrs.) = (no. of steps x time taken for each iteration by the GPU)/3600

D

druhl•9/1/24, 5:28 AM

What I do not understand (yet!), however, is:
1) what difference it makes if you use 3000 steps versus, say 7000 steps (increase the epoch for the same no. of input images. My assumption is it takes more time, but the quality will be better to a certain extent, before it gives diminishing returns and then peaks).
2) Use less images & more epoch, versus more images and less epoch (However, the no. of steps is maintained the same. Eg. 15x200=3000, and 50x60=also3000. Here, I make no assumption and would like to hear from Doc).

D

dxqb•9/1/24, 6:45 AM

1) largely correct, but there is no peak, there is a cliff. you can overfit diffusion models quite easily

D

dxqb•9/1/24, 6:50 AM

2) there is no clear answer, it's an art. Less images tend to give you an inflexible result, more images tend to be harder to train until the model picks up the details. You often need more steps in total for many images than for a few, meaning not reduce the epochs as much as you have increased the number of images. Larger batch sizes can help with (many) more images. Adjustments in LR might be necessary.

D

dxqb•9/1/24, 6:52 AM

and to make it even less predictable, it depends on the dataset, especially how similar the images are

D

druhl•9/1/24, 6:54 AM

Yes, think I understand.
Good to see I got the basics right.

D

druhl•9/1/24, 6:54 AM

Thanks for explaining further.

N

Neuromasters•9/1/24, 7:54 AM

it writes errors, do I have to put in an experimental file?

T

Thereiver•9/1/24, 8:51 AM

Hey, does anyone have a prodigy config for flux? Back in the day we’d set the warmup steps to 100% and have loras ready in 30-45 mins or so depending on the amount of images in the dataset.

I like the results that your rank 3/4 config provides, it’s really high quality. But I’d need something a bit faster than 35 hrs for a 300 image dataset - any tips would be really appreciated

MMaxivi i think for linux xformers works fine because of the torch version

F

Furkan Gözükara SECourses•9/1/24, 10:14 AM

you were right i see improvment in the training

F

Furkan Gözükara SECourses•9/1/24, 10:14 AM

looks like bmaltais fixed something for sure

F

Furkan Gözükara SECourses•9/1/24, 10:14 AM

prev vs new