what gpu which config how much vram used before starting training
what gpu which config how much vram used before starting training
GThe A6000 was the same, both loaded with your config, I used Dataset Preperation, I have a LORA Folder inside, Model, Log, Image folder, inside that folder 1_ohwx art inside there 10,000 images, .txt files + the other file it makes
Fye i know i mean
Fi wonder if your machine broken
Fcan you try with 2500 images
GI am renting these on Masscompute
Fbut
Fperhaps error is
Gtoo many images problem?
Fcaching part
Fyou are at the caching part right?
G
Git crashes here
Fok lets do this
Ffirst start as single gpu
Fwait until it starts training
Fthen stop training
Fstart as multi gpu
Fi think caching part is the problem
Fonce cached it wont recache
Gso let it cache with 1 gpu, start training, stop it, reconfigure for 4 GPU
Fi plan to train 20k images
Fbut didnt have time yet
Git must be the 10K images
Fwhat i feel is
Fat caching part
Fit fails
Fso once cached
Fi think yours will work
Fi am testing torch 2.4.1 on windows
GOK I shutoff the L40S, luckily there is a 4GPU A6000 now
Gi took it and will test
Fsure
Fgood idea
Fonce you make it work
Fyou can move to l40s
Fyou can even move cached files
Fit shouldnt recache on l40s
GOK I will copy the Lora directory over onced cached
N3 rank
4090 gpu loсal
4090 gpu loсal
Fit shoudl work
Fworking or not?
Noperate
Fnice
Fworking good
Fif you use rank 4 will be faster
Fi am testing torch 2.4.1 right now
Fit may boost even more
Flooks like 2.4.1 still doesnt fix :/
Fok torch 2.5 still best