so let it cache with 1 gpu, start training, stop it, reconfigure for 4 GPU
so let it cache with 1 gpu, start training, stop it, reconfigure for 4 GPU
Fi plan to train 20k images
Fbut didnt have time yet
Git must be the 10K images
Fwhat i feel is
Fat caching part
Fit fails
Fso once cached
Fi think yours will work
Fi am testing torch 2.4.1 on windows
GOK I shutoff the L40S, luckily there is a 4GPU A6000 now
Gi took it and will test
Fsure
Fgood idea
Fonce you make it work
Fyou can move to l40s
Fyou can even move cached files
Fit shouldnt recache on l40s
GOK I will copy the Lora directory over onced cached
N3 rank
4090 gpu loсal
4090 gpu loсal
Fit shoudl work
Fworking or not?
Noperate
Fnice
Fworking good
Fif you use rank 4 will be faster
Fi am testing torch 2.4.1 right now
Fit may boost even more
Flooks like 2.4.1 still doesnt fix :/
Fok torch 2.5 still best
Ffor windows
Fimproves both rank 3,4,5,6 and all 
Fi am gonna test something else now a new stuff
Frtx 3090 and rtx 3060 faster ranks
Fmin speeds on windows
WIs there a way to change it to use 512x512 images as the source?
Gstill doesnt work, 1 GPU dies in same spot
GI wasted 20$ on multiple instances, I will wait until you test 10,000 IMAGES with Captions and figure out the correct settings
Git does not work
M@Dr. Furkan Gözükara t5 training is out
Mgetting way better results now with captions
M
Fjust change 1024 to 512 on gradio
Fyou should post it as error on kohya meanwhile
Fmaybe it is related to him
Fye i saw
Fbmaltais also merged nice
Fcurrently uses same LR i presume
Myou can't cache text encoder on bmaltais