Software Engineering Courses (SECourses)•17mo ago

fully distant shot 😄

fully distant shot

VVs12 lora is sort of crap because of the different alphas and conv needed

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

i like fine tuning too. but first started lora and sadly it doesnt work with 12 gb config

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

it uses 18 gb

FFurkan Gözükara SECourses fully distant shot 😄

Vs12•8/21/24, 11:53 PM

Mafia boss

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

i tell this to kohya he says working . i have shown full config toml everything. tested on windows and ubuntu

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

uses 18 gb with half model split training

FFurkan Gözükara SECourses i like fine tuning too. but first started lora and sadly it doesnt work with 12 ...

Vs12•8/21/24, 11:53 PM

when training on the fp8 model probably less vram

Vs12•8/21/24, 11:53 PM

when available

VVs12 when training on the fp8 model probably less vram

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

it is fp8 plus half model

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

still uses 18 gb

Vs12•8/21/24, 11:54 PM

are you training on the full 23gb model?

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

VVs12 are you training on the full 23gb model?

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

i tested both fp8 model and fp16 model but it doesnt matter the precision i set is fp8 mode

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

the vram usage is very erratic as shown above

Vs12•8/21/24, 11:55 PM

hopefully one trainer starts working on it

Vs12•8/21/24, 11:55 PM

have your tried the bitsandbytes flux model?

VVs12 have your tried the bitsandbytes flux model?

Furkan Gözükara SECoursesOP•8/21/24, 11:56 PM

what you mean by that

Furkan Gözükara SECoursesOP•8/21/24, 11:56 PM

accelerator has fp8 mode but it doesnt work with kohya :d

Vs12•8/21/24, 11:56 PM

https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors

flux1-dev-bnb-nf4-v2.safetensors · lllyasviel/flux1-dev-bnb-nf4 at ...

VVs12 https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-...

Furkan Gözükara SECoursesOP•8/21/24, 11:56 PM

i dont think they are trainable but didnt test

Vs12•8/21/24, 11:57 PM

you should upgrade to torch 2.4.0 and vision 19.0

Vs12•8/21/24, 11:57 PM

it uses the new cudnn 9.01

Vs12•8/21/24, 11:57 PM

i think faster

Vs12•8/21/24, 11:57 PM

torch==2.4.0+cu124 --index-url https://download.pytorch.org/whl/cu124
torchvision==0.19.0+cu124 --index-url https://download.pytorch.org/whl/cu124

FFurkan Gözükara SECourses i dont think they are trainable but didnt test

Vs12•8/21/24, 11:59 PM

havent seen training on it yet, so not sure

Furkan Gözükara SECoursesOP•8/21/24, 11:59 PM

let me test these to see vram usage

Vs12•8/22/24, 12:00 AM

apprently 50% to 100% speed boost

Vs12•8/22/24, 12:01 AM

https://docs.nvidia.com/deeplearning/cudnn/latest/release-notes.html#cudnn-9-0-0

Vs12•8/22/24, 12:01 AM

but i think only for fp16 and bf16

VVs12 but i think only for fp16 and bf16

Furkan Gözükara SECoursesOP•8/22/24, 12:01 AM

nice

Furkan Gözükara SECoursesOP•8/22/24, 12:01 AM

i am gonna test later

Furkan Gözükara SECoursesOP•8/22/24, 12:01 AM

i dont want to take risk of breaking quality :d

Vs12•8/22/24, 12:02 AM

i havent noticed quality difference

but you can try later

VVs12 i havent noticed quality difference 🙂 but you can try later

Furkan Gözükara SECoursesOP•8/22/24, 12:03 AM

did you see speed diff?

Furkan Gözükara SECoursesOP•8/22/24, 12:03 AM

configs do make huge difference

Furkan Gözükara SECoursesOP•8/22/24, 12:04 AM

so many people blindly using and not testing

FFurkan Gözükara SECourses did you see speed diff?

Vs12•8/22/24, 12:04 AM

i think so, i am on 4090 and it feels much faster, i will try some more tests and compare speeds for you

KGTachi•8/22/24, 12:05 AM

@Dr. Furkan Gözükara are you planning on making joy caption ?

VVs12 i think so, i am on 4090 and it feels much faster, i will try some more tests an...

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

nice

KKGTachi @Dr. Furkan Gözükara are you planning on making joy caption ?

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

yep

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

on my next list

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

hopefully today or tomorrow

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

please remind me again tomorrow

Dazzastrous [4090]•8/22/24, 12:05 AM

yeh its really good

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

i will add batch caption too and cloud as well

KGTachi•8/22/24, 12:05 AM

Thank you

I will subscribe Patreon . Saw your video on YouTube

KKGTachi Thank you 🙏 I will subscribe Patreon . Saw your video on YouTube

Furkan Gözükara SECoursesOP•8/22/24, 12:06 AM

awesome ty

Dazzastrous [4090]•8/22/24, 12:06 AM

did ur dataset include pics of you not wearing specs?

DDazzastrous [4090]did ur dataset include pics of you not wearing specs?

�

𝔸𝖨ΞӾ•8/22/24, 12:13 AM

I really think it has something to do with the thin frames that are almost invisible around the lenses and that how the sides are black too and blend in with his hair color but I dunno I'm no expert

DDazzastrous [4090]did ur dataset include pics of you not wearing specs?

Furkan Gözükara SECoursesOP•8/22/24, 12:17 AM

dataset

Furkan Gözükara SECoursesOP•8/22/24, 12:19 AM

flux lora training with 9 gb

fully distant shot 😄

Similar Threads