Software Engineering Courses (SECourses)•17mo ago

reg images improves environment but likeliness is gone in some cases - 2225 step

FFurkan Gözükara SECourses reg images improves environment but likeliness is gone in some cases - 2225 step

Vs12•8/21/24, 11:49 PM

drop your prior weight loss to 0.10

Vs12•8/21/24, 11:49 PM

and more steps

VVs12 drop your prior weight loss to 0.10

Furkan Gözükara SECoursesOP•8/21/24, 11:50 PM

nice idea should test

Vs12•8/21/24, 11:51 PM

reg images not reccomended for flux, but 0.10 and lower prior loss and higher lr ideal

VVs12 reg images not reccomended for flux, but 0.10 and lower prior loss and higher lr...

Furkan Gözükara SECoursesOP•8/21/24, 11:51 PM

just testing atm. so far got great results and getting even better with finding better params

FFurkan Gözükara SECourses just testing atm. so far got great results and getting even better with finding ...

Vs12•8/21/24, 11:52 PM

same as well, been getting amazing results with finetune/dreambooth as well

Vs12•8/21/24, 11:52 PM

lora is sort of crap because of the different alphas and conv needed

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

fully distant shot

VVs12 lora is sort of crap because of the different alphas and conv needed

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

i like fine tuning too. but first started lora and sadly it doesnt work with 12 gb config

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

it uses 18 gb

FFurkan Gözükara SECourses fully distant shot 😄

Vs12•8/21/24, 11:53 PM

Mafia boss

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

i tell this to kohya he says working . i have shown full config toml everything. tested on windows and ubuntu

Furkan Gözükara SECoursesOP•8/21/24, 11:53 PM

uses 18 gb with half model split training

FFurkan Gözükara SECourses i like fine tuning too. but first started lora and sadly it doesnt work with 12 ...

Vs12•8/21/24, 11:53 PM

when training on the fp8 model probably less vram

Vs12•8/21/24, 11:53 PM

when available

VVs12 when training on the fp8 model probably less vram

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

it is fp8 plus half model

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

still uses 18 gb

Vs12•8/21/24, 11:54 PM

are you training on the full 23gb model?

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

VVs12 are you training on the full 23gb model?

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

i tested both fp8 model and fp16 model but it doesnt matter the precision i set is fp8 mode

Furkan Gözükara SECoursesOP•8/21/24, 11:54 PM

the vram usage is very erratic as shown above

Vs12•8/21/24, 11:55 PM

hopefully one trainer starts working on it

Vs12•8/21/24, 11:55 PM

have your tried the bitsandbytes flux model?

VVs12 have your tried the bitsandbytes flux model?

Furkan Gözükara SECoursesOP•8/21/24, 11:56 PM

what you mean by that

Furkan Gözükara SECoursesOP•8/21/24, 11:56 PM

accelerator has fp8 mode but it doesnt work with kohya :d

Vs12•8/21/24, 11:56 PM

https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors

flux1-dev-bnb-nf4-v2.safetensors · lllyasviel/flux1-dev-bnb-nf4 at ...

VVs12 https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-...

Furkan Gözükara SECoursesOP•8/21/24, 11:56 PM

i dont think they are trainable but didnt test

Vs12•8/21/24, 11:57 PM

you should upgrade to torch 2.4.0 and vision 19.0

Vs12•8/21/24, 11:57 PM

it uses the new cudnn 9.01

Vs12•8/21/24, 11:57 PM

i think faster

Vs12•8/21/24, 11:57 PM

torch==2.4.0+cu124 --index-url https://download.pytorch.org/whl/cu124
torchvision==0.19.0+cu124 --index-url https://download.pytorch.org/whl/cu124

FFurkan Gözükara SECourses i dont think they are trainable but didnt test

Vs12•8/21/24, 11:59 PM

havent seen training on it yet, so not sure

Furkan Gözükara SECoursesOP•8/21/24, 11:59 PM

let me test these to see vram usage

Vs12•8/22/24, 12:00 AM

apprently 50% to 100% speed boost

Vs12•8/22/24, 12:01 AM

https://docs.nvidia.com/deeplearning/cudnn/latest/release-notes.html#cudnn-9-0-0

Vs12•8/22/24, 12:01 AM

but i think only for fp16 and bf16

VVs12 but i think only for fp16 and bf16

Furkan Gözükara SECoursesOP•8/22/24, 12:01 AM

nice

Furkan Gözükara SECoursesOP•8/22/24, 12:01 AM

i am gonna test later

Furkan Gözükara SECoursesOP•8/22/24, 12:01 AM

i dont want to take risk of breaking quality :d

Vs12•8/22/24, 12:02 AM

i havent noticed quality difference

but you can try later

VVs12 i havent noticed quality difference 🙂 but you can try later

Furkan Gözükara SECoursesOP•8/22/24, 12:03 AM

did you see speed diff?

Furkan Gözükara SECoursesOP•8/22/24, 12:03 AM

configs do make huge difference

Furkan Gözükara SECoursesOP•8/22/24, 12:04 AM

so many people blindly using and not testing

FFurkan Gözükara SECourses did you see speed diff?

Vs12•8/22/24, 12:04 AM

i think so, i am on 4090 and it feels much faster, i will try some more tests and compare speeds for you

KGTachi•8/22/24, 12:05 AM

@Dr. Furkan Gözükara are you planning on making joy caption ?

VVs12 i think so, i am on 4090 and it feels much faster, i will try some more tests an...

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

nice

KKGTachi @Dr. Furkan Gözükara are you planning on making joy caption ?

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

yep

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

on my next list

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

hopefully today or tomorrow

Furkan Gözükara SECoursesOP•8/22/24, 12:05 AM

please remind me again tomorrow

reg images improves environment but likeliness is gone in some cases - 2225 step

Similar Threads

Similar Threads

Similar Threads