Software Engineering Courses (SECourses)•16mo ago

Style aligned testing and working.

PPelolisu Style aligned testing and working.

Furkan Gözükara SECourses•9/21/24, 1:01 AM

awesome

Furkan Gözükara SECourses•9/21/24, 2:52 AM

i am gonna fine tune my 256 images dataset lets see best quality :d

Furkan Gözükara SECourses•9/21/24, 2:53 AM

Furkan Gözükara SECourses•9/21/24, 2:58 AM

200 epochs taking like 88 hours on a single gpu but probably will be too overtrained. i expect like 70 80 epoch best

Furkan Gözükara SECourses•9/21/24, 2:58 AM

rtx 5090 if comes with 32 gb it would train like in 15 hours i expect

PelolisuOP•9/21/24, 3:02 AM

the amount of tokens in captions increase the time ?

Furkan Gözükara SECourses•9/21/24, 3:02 AM

no impact

Furkan Gözükara SECourses•9/21/24, 3:02 AM

this dataset is 256 images 200 epoch

PelolisuOP•9/21/24, 3:04 AM

if is 128 images with 2 repeat the time is the same ?

Furkan Gözükara SECourses•9/21/24, 3:05 AM

yes would be samer

PelolisuOP•9/21/24, 3:06 AM

with batch size two maybe can reduce to a half ?

PPelolisu with batch size two maybe can reduce to a half ?

Furkan Gözükara SECourses•9/21/24, 3:07 AM

nope

Furkan Gözükara SECourses•9/21/24, 3:08 AM

step count will be half but it brings like 20% speed up

Furkan Gözükara SECourses•9/21/24, 3:08 AM

batch size 3 brings almost 0%

Furkan Gözükara SECourses•9/21/24, 3:08 AM

if it was fitting into 48 gb gpus when doing multi gpu training

Furkan Gözükara SECourses•9/21/24, 3:08 AM

that brings almost 100% speed up

Furkan Gözükara SECourses•9/21/24, 3:08 AM

i trained model with loras that way used 8x gpu

Furkan Gözükara SECourses•9/21/24, 3:10 AM

status of general fine tuning

Furkan Gözükara SECourses•9/21/24, 3:10 AM

20800 steps is first checkpoint hopefully

PelolisuOP•9/21/24, 3:15 AM

in this case, the increase in memory consumption increases

PelolisuOP•9/21/24, 3:15 AM

I suspect that part of the blame for the slowness in training is poorly designed algorithms. But that's what we have for today haha.

Ddxqb what does flow shift do? OT doesn't have that

WetOnTheWater•9/21/24, 8:22 AM

I have no idea tbh

Ddxqb agree on 128. Are you using dropout for Lora?

WetOnTheWater•9/21/24, 8:22 AM

Is dropout supported for flux in kohya yet?

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 8:27 AM

@Dr. Furkan Gözükara Is multi gpu training not supported on windows?

PPelolisu I suspect that part of the blame for the slowness in training is poorly designed...

Furkan Gözükara SECourses•9/21/24, 11:34 AM

very likely

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 @Dr. Furkan Gözükara Is multi gpu training not supported on windows?

Furkan Gözükara SECourses•9/21/24, 11:34 AM

i never tried

FFurkan Gözükara SECourses i never tried

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 11:35 AM

I get this error - Redirects are currently not supported in Windows or MacOs.
when multi gpu clicked

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 I get this error - Redirects are currently not supported in Windows or MacOs. wh...

Furkan Gözükara SECourses•9/21/24, 12:23 PM

maybe not supported

Furkan Gözükara SECourses•9/21/24, 2:01 PM

Installing kohya on 2x A100 80 GB machine on RunPod took 30 freaking minutes

FFurkan Gözükara SECourses Installing kohya on 2x A100 80 GB machine on RunPod took 30 freaking minutes

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:03 PM

Are you using secure cloud?

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 Are you using secure cloud?

Furkan Gözükara SECourses•9/21/24, 2:03 PM

yes

FFurkan Gözükara SECourses yes

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:04 PM

Sometimes you get these buggy pods. I just delete that pod and start a new one

Furkan Gözükara SECourses•9/21/24, 2:05 PM

they just suck it happens to frequently

Furkan Gözükara SECourses•9/21/24, 2:05 PM

their biggest advantage is permanent storage but this slowness killing it

FFurkan Gözükara SECourses they just suck it happens to frequently

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:07 PM

Their ui is much superior to massed compute and I don't get this problem very often and most of the time I am using 8X l40s which works super fast

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 Their ui is much superior to massed compute and I don't get this problem very of...

Furkan Gözükara SECourses•9/21/24, 2:08 PM

8x l40s of course would be fast lol

Furkan Gözükara SECourses•9/21/24, 2:08 PM

it is newest machine

FFurkan Gözükara SECourses it is newest machine

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:09 PM

oh yes, I forgot that

Furkan Gözükara SECourses•9/21/24, 2:46 PM

20 epoch fine tuning 256 images

Furkan Gözükara SECourses•9/21/24, 2:47 PM

it can do cartoon atm but not very resemblaing

Furkan Gözükara SECourses•9/21/24, 2:47 PM

Furkan Gözükara SECourses•9/21/24, 2:48 PM

it can do beard amazing

Furkan Gözükara SECourses•9/21/24, 2:48 PM

Furkan Gözükara SECourses•9/21/24, 2:48 PM

Furkan Gözükara SECourses•9/21/24, 2:57 PM

this bleeding issue of flux so annoying

Furkan Gözükara SECourses•9/21/24, 2:57 PM

20e poch photo of barrack obama

1457-photo20of20barack20obama-FT_256_images-000020-1690245631.png

Dazzastrous [4090]•9/21/24, 2:58 PM

FFurkan Gözükara SECourses 20e poch photo of barrack obama

druhl•9/21/24, 3:09 PM

That's Furrack Obakara

druhl•9/21/24, 3:09 PM

Style aligned testing and working.

Similar Threads

Similar Threads

Similar Threads