Software Engineering Courses (SECourses)•16mo ago

with batch size two maybe can reduce to a half ?

PPelolisu with batch size two maybe can reduce to a half ?

Furkan Gözükara SECourses•9/21/24, 3:07 AM

nope

Furkan Gözükara SECourses•9/21/24, 3:08 AM

step count will be half but it brings like 20% speed up

Furkan Gözükara SECourses•9/21/24, 3:08 AM

batch size 3 brings almost 0%

Furkan Gözükara SECourses•9/21/24, 3:08 AM

if it was fitting into 48 gb gpus when doing multi gpu training

Furkan Gözükara SECourses•9/21/24, 3:08 AM

that brings almost 100% speed up

Furkan Gözükara SECourses•9/21/24, 3:08 AM

i trained model with loras that way used 8x gpu

Furkan Gözükara SECourses•9/21/24, 3:10 AM

status of general fine tuning

Furkan Gözükara SECourses•9/21/24, 3:10 AM

20800 steps is first checkpoint hopefully

PelolisuOP•9/21/24, 3:15 AM

in this case, the increase in memory consumption increases

PelolisuOP•9/21/24, 3:15 AM

I suspect that part of the blame for the slowness in training is poorly designed algorithms. But that's what we have for today haha.

Ddxqb what does flow shift do? OT doesn't have that

WetOnTheWater•9/21/24, 8:22 AM

I have no idea tbh

Ddxqb agree on 128. Are you using dropout for Lora?

WetOnTheWater•9/21/24, 8:22 AM

Is dropout supported for flux in kohya yet?

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 8:27 AM

@Dr. Furkan Gözükara Is multi gpu training not supported on windows?

PPelolisu I suspect that part of the blame for the slowness in training is poorly designed...

Furkan Gözükara SECourses•9/21/24, 11:34 AM

very likely

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 @Dr. Furkan Gözükara Is multi gpu training not supported on windows?

Furkan Gözükara SECourses•9/21/24, 11:34 AM

i never tried

FFurkan Gözükara SECourses i never tried

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 11:35 AM

I get this error - Redirects are currently not supported in Windows or MacOs.
when multi gpu clicked

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 I get this error - Redirects are currently not supported in Windows or MacOs. wh...

Furkan Gözükara SECourses•9/21/24, 12:23 PM

maybe not supported

Furkan Gözükara SECourses•9/21/24, 2:01 PM

Installing kohya on 2x A100 80 GB machine on RunPod took 30 freaking minutes

FFurkan Gözükara SECourses Installing kohya on 2x A100 80 GB machine on RunPod took 30 freaking minutes

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:03 PM

Are you using secure cloud?

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 Are you using secure cloud?

Furkan Gözükara SECourses•9/21/24, 2:03 PM

yes

FFurkan Gözükara SECourses yes

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:04 PM

Sometimes you get these buggy pods. I just delete that pod and start a new one

Furkan Gözükara SECourses•9/21/24, 2:05 PM

they just suck it happens to frequently

Furkan Gözükara SECourses•9/21/24, 2:05 PM

their biggest advantage is permanent storage but this slowness killing it

FFurkan Gözükara SECourses they just suck it happens to frequently

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:07 PM

Their ui is much superior to massed compute and I don't get this problem very often and most of the time I am using 8X l40s which works super fast

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 Their ui is much superior to massed compute and I don't get this problem very of...

Furkan Gözükara SECourses•9/21/24, 2:08 PM

8x l40s of course would be fast lol

Furkan Gözükara SECourses•9/21/24, 2:08 PM

it is newest machine

FFurkan Gözükara SECourses it is newest machine

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/21/24, 2:09 PM

oh yes, I forgot that

Furkan Gözükara SECourses•9/21/24, 2:46 PM

20 epoch fine tuning 256 images

Furkan Gözükara SECourses•9/21/24, 2:47 PM

it can do cartoon atm but not very resemblaing

Furkan Gözükara SECourses•9/21/24, 2:47 PM

Furkan Gözükara SECourses•9/21/24, 2:48 PM

it can do beard amazing

Furkan Gözükara SECourses•9/21/24, 2:48 PM

Furkan Gözükara SECourses•9/21/24, 2:48 PM

Furkan Gözükara SECourses•9/21/24, 2:57 PM

this bleeding issue of flux so annoying

Furkan Gözükara SECourses•9/21/24, 2:57 PM

20e poch photo of barrack obama

1457-photo20of20barack20obama-FT_256_images-000020-1690245631.png

Dazzastrous [4090]•9/21/24, 2:58 PM

FFurkan Gözükara SECourses 20e poch photo of barrack obama

druhl•9/21/24, 3:09 PM

That's Furrack Obakara

druhl•9/21/24, 3:09 PM

Furkan Gözükara SECourses•9/21/24, 3:32 PM

new public article

Furkan Gözükara SECourses•9/21/24, 3:32 PM

https://www.patreon.com/posts/112494682?pr=true

Patreon

Multi-GPU FLUX Full Fine Tuning Experiments and Requirements on Run...

Get more from SECourses: Tutorials, Guides, Resources, Training, FLUX, MidJourney, Voice Clone, TTS, ChatGPT, GPT, LLM, Scripts by Furkan Gözü on Patreon

daroth•9/21/24, 5:19 PM

at the moment nothing points to the 5090 having more than 28gigs of vram, even that is hearsay

Ddaroth at the moment nothing points to the 5090 having more than 28gigs of vram, even t...

Furkan Gözükara SECourses•9/21/24, 5:20 PM

Ye so shameless

Furkan Gözükara SECourses•9/21/24, 5:28 PM

500 steps left first checkpoint

daroth•9/21/24, 5:42 PM

thats gonna take a while

Ddaroth thats gonna take a while 🙂

Furkan Gözükara SECourses•9/21/24, 6:16 PM

Maxivi•9/21/24, 6:20 PM

@Dr. Furkan Gözükara any update on the fp8 for dreambooth Furak Obama?

Maxivi•9/21/24, 6:31 PM

@Dr. Furkan Gözükara do you get this issue when extracting lora

flux_extract_lora.py:68
31%|█████████████████████████████████████████████████████████████████████████████ | 93/304 [01:57<09:26, 1.86s/it]
Traceback (most recent call last):
File "/workspace/kohya_ss/sd-scripts/networks/flux_extract_lora.py", line 219, in <module>
svd(**vars(args))
File "/workspace/kohya_ss/sd-scripts/networks/flux_extract_lora.py", line 87, in svd
value_t = f_tuned.get_tensor(key)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 249, in get_tensor
return self._deserialize_tensor(tensor_bytes, metadata)
File "/workspace/kohya_ss/sd-scripts/library/utils.py", line 265, in _deserialize_tensor
byte_tensor = torch.frombuffer(tensor_bytes, dtype=torch.uint8)
ValueError: both buffer length (0) and count (-1) must not be 0

MMaxivi @Dr. Furkan Gözükara do you get this issue when extracting lora ...

Furkan Gözükara SECourses•9/21/24, 6:36 PM

nope

MMaxivi @Dr. Furkan Gözükara any update on the fp8 for dreambooth Furak Obama?

Furkan Gözükara SECourses•9/21/24, 6:36 PM

nope too

Furkan Gözükara SECourses•9/21/24, 6:36 PM

testing first checkpoint

with batch size two maybe can reduce to a half ?

Similar Threads

Similar Threads

Similar Threads