Oh my God. My computer froze at 48% of the Lora training. Can I somehow resume the training? Or do I
Oh my God. My computer froze at 48% of the Lora training. Can I somehow resume the training? Or do I have to start over?
CNow i know we need MARK some option in Koya to be able later resume training
CUnder PArameters - Advence
C
CYou should mention about this Boss.
Fyou can continue from last checkpoint lora too
F
Ffor batch size 1 L40S is good choice others are not worth very much
�Oh, L40S will be faster, but quality output will be same as a6000 yes?
Fbut A6000 is the most price / performance
Fon massed compute
�Oh, so if I want fastest possible training, it's the h100.
�rented 1 comp for a month with a6000, at the end of the month, it was so slow, totally unresponsive, i was not even able to do a restart, so had to delete the deployment and redeploy. Maybe I should have restarted the machine more during that month, or maybe you're not supposed to rent it for that long in one run.
Fwas it runpod?
�massed compute
Fthey rarely have bad machines
Fif you report machine id here
Fwe can ping the owners
�ok thank you, I probably can't find that id now, when it's deleted?
Fyou cant. when that happens report here and ping me :d
�thank you! btw, should I still use your downloader zip file for kohya when using massed compute? or just run kohya update then run kohya directly from massed compute?
Fuse my zip files
Fbest and most up to date
�excellent, was just about to start, 91 character training pics. 1 character. 200 epochs save at every 25 should be ok? my last character train with 66 pics, when comparing epocs for realism, 25, 50, 75, 100, 125, the last one, epoch 125 was the best, with 100 being the 2nd best.
F200 could be overfit for 91 images
�so 150?
F150 good
Fand compare checkpoints
�yes, save every 25
Fbut if you wna to be 100% sure 200 way to go
�sure, for research then
�200, save every 25, then do the amazing prompts and grid you taught us 
thank you for the quick reply, no one works harder than you in AI, and we are grateful!
thank you for the quick reply, no one works harder than you in AI, and we are grateful!Fyou can also save like every 20 or 15 depending on storage
�how much faster is the h100 compared to the a100 to complete fine tuning?
�i'm taking the h100 now, can try a100 next time.
Fbut this looks wrong
Fwait
Fok it looks true
FL40s and H100 same
Fso L40s would be faster than a100
Factually best case looks like comparing all 3
�ha, so more flops means higher speed. how about dual h100, would that make it faster or not?
�The speed of H100 GPU , 2.54s/it .
Freally good
Fbut dont worth it i think
FA6000 gets 6.3 second
Fso it is like 2.4 times faster