Software Engineering Courses (SECourses)•17mo ago

I was able to train at 1.2 it/s with gradient checkpoint off and tweaking some other options, I thi

I was able to train at 1.2 it/s with gradient checkpoint off and tweaking some other options, I thing full bf16 training experimental and some other things I dont remember right now

dsienraOP•8/20/24, 3:12 PM

with sdxl off course

NNeuromasters what are the best ones for Flux right now?

Furkan Gözükara SECourses•8/20/24, 3:14 PM

havent tested yet just using default :d

Ddsienra I was able to train at 1.2 it/s with gradient checkpoint off and tweaking some ...

Furkan Gözükara SECourses•8/20/24, 3:15 PM

1.2 it is good speed for RTX 4090 and SDXL

FFurkan Gözükara SECourses 1.2 it is good speed for RTX 4090 and SDXL

dsienraOP•8/20/24, 3:18 PM

may be It was 1.4/it but oround that speed

dsienraOP•8/20/24, 3:18 PM

my card is a 3090

dsienraOP•8/20/24, 3:27 PM

I hade slightly better speed on windows 10 0.1 it/s better

Furkan Gözükara SECourses•8/20/24, 3:29 PM

ah you must be using xformers

Furkan Gözükara SECourses•8/20/24, 3:30 PM

i think it reduces quality slightly

dsienraOP•8/20/24, 3:35 PM

yes I think you right

Neuromasters•8/20/24, 3:36 PM

What's the new parameter

NNeuromasters What's the new parameter

Furkan Gözükara SECourses•8/20/24, 3:40 PM

probably for turbo model

dsienraOP•8/20/24, 3:51 PM

dsienraOP•8/20/24, 3:52 PM

1e_04_fp8_bf16_accelerate_full_bf16_32_rank config

dsienraOP•8/20/24, 3:52 PM

something is not right

dsienraOP•8/20/24, 3:52 PM

is consuming too much vram

dsienraOP•8/20/24, 3:54 PM

dsienraOP•8/20/24, 3:59 PM

one thing I saw yesterday was that no matter the lora rank I use the vram usage was the same

dsienraOP•8/20/24, 4:01 PM

lora rank 32

dsienraOP•8/20/24, 4:07 PM

lora rank 128 is consuming less vram??? what??

dsienraOP•8/20/24, 4:08 PM

must be a bug

Ddsienra lora rank 128 is consuming less vram??? what??

Furkan Gözükara SECourses•8/20/24, 4:18 PM

lora rank impact on VRAM is very minimal i tested

Ddsienra lora rank 128 is consuming less vram??? what??

Furkan Gözükara SECourses•8/20/24, 4:18 PM

how much vram usage you have before starting the training?

dsienraOP•8/20/24, 4:22 PM

0.6

dsienraOP•8/20/24, 4:22 PM

I can turn off the monitors and get 0.4 gb

Ddsienra 0.6

Furkan Gözükara SECourses•8/20/24, 4:23 PM

0.6 should work but still using some extra. i wonder if due to operating system

Furkan Gözükara SECourses•8/20/24, 4:23 PM

i tested vram usages on ubuntu

dsienraOP•8/20/24, 4:23 PM

but yesterday worked

Furkan Gözükara SECourses•8/20/24, 4:23 PM

interesting

Furkan Gözükara SECourses•8/20/24, 4:23 PM

are you using latest version?

Furkan Gözükara SECourses•8/20/24, 4:23 PM

of kohya

Ddsienra Is training on my rtx 3090 but 9 times slower than sdxl, lora rank 8 1024x1024 r...

dsienraOP•8/20/24, 4:24 PM

I cant remenber what was diferent

dsienraOP•8/20/24, 4:25 PM

it was rank 8

dsienraOP•8/20/24, 4:26 PM

now is working rank 8

dsienraOP•8/20/24, 4:28 PM

dsienraOP•8/20/24, 4:29 PM

for Network Rank (Dimension) 8 I need to set Convolution Rank (Dimension) to 8 too?

dsienraOP•8/20/24, 4:30 PM

and what about Network Alpha and Convolution Alpha?

FFurkan Gözükara SECourses are you using latest version?

dsienraOP•8/20/24, 4:31 PM

yes I did a git pull before training

dsienraOP•8/20/24, 5:04 PM

I'm going to test your lowest_vram config now

dsienraOP•8/20/24, 5:53 PM

Now is working with your 1e_04_fp8_bf16_accelerate_full_bf16_32_rank config, I just restarted the PC and enabled buckets not all my images are 1024x1024 is working with great speed

dsienraOP•8/20/24, 6:01 PM

that was with my monitors turned off but I need to use my pc for other tasks, using the pc the speed decreased

Ddsienra Now is working with your 1e_04_fp8_bf16_accelerate_full_bf16_32_rank config, I j...

�

𝔸𝖨ΞӾ•8/20/24, 6:03 PM

Is 200 epochs per image really necessary tho? 387 images in your dataset and training will be finished like in 4 days lol

dsienraOP•8/20/24, 6:05 PM

hehe yes

dsienraOP•8/20/24, 6:06 PM

I can stop it earlier and test

dsienraOP•8/20/24, 6:06 PM

but yes is too slow

Ddsienra hehe yes

�

𝔸𝖨ΞӾ•8/20/24, 6:08 PM

Ok because I was testing Dr's 128 rank config earlier and was wondering why it was set at 200 epochs, not sure how many images he had so I was just curious why. I'm still learning btw

Dazzastrous [4090]•8/20/24, 6:26 PM

any idea why my comp completely freezes when trying to install Dlib in conda?

Ddsienra and what about Network Alpha and Convolution Alpha?

Furkan Gözükara SECourses•8/20/24, 7:09 PM

Convolution Rank is for other lora types as far as i know

Furkan Gözükara SECourses•8/20/24, 7:10 PM

make network rank 8 and alpha 8

Ddsienra Now is working with your 1e_04_fp8_bf16_accelerate_full_bf16_32_rank config, I j...

Furkan Gözükara SECourses•8/20/24, 7:10 PM

great

�𝔸𝖨ΞӾ Is 200 epochs per image really necessary tho? 387 images in your dataset and tra...

Furkan Gözükara SECourses•8/20/24, 7:10 PM

nope i am doing big test

I was able to train at 1.2 it/s with gradient checkpoint off and tweaking some other options, I thi

Similar Threads