Do you meat at training? At inference L40S is only 8% slower but 0.3 second new prompt delay opposed
Do you meat at training? At inference L40S is only 8% slower but 0.3 second new prompt delay opposed to >3s on 5090
Tso for practical purposes L40S is still better
Fi think it depends if you are constantly changing prompt yes
Fbut L40S is 8000+ USD 
Fand 5090 is at its slowest performance
Fit will only get better we dont even have official pytorch for windows
P@Furkan Gözükara SECourses Hello I just started training with the 24 GB flux dev model but my storage was less only 50gb so i think it can create only 2 checkpoints but I checked the training status and the second checkpoint epoch has already strated even before checkpoint one epochs are totally complete.How to save one checkpoint to huggingface if the second one has also started simultaneously
P@Furkan Gözükara SECourses and also once the storage is full what error will I get and how to resume the training
P@Furkan Gözükara SECourses actually I have set 4 checkpoints instead of 2
Fi never could make it save to hugging face before automatically from kohya
Fonce storage is full i dont remember behaviour sadly
P@Furkan Gözükara SECourses So can i just stop this process and increase the storage for the same gpu instance and the again start training
Fon runpod yes
PI am on masscompute
Fyou cant there
Fget L40 machine it has way more storage
For multiple A6000 GPU
Fby the way
Fyou can delete a lot of apps there
Fall will increase storage
Fyou can delete all except kohya
PAs you had said in the long tutorial that checkpoints can be saved midway so I proceeded with 4 checkpoints 
Fyou can. but you have to monitor
Fas soon as saved upload to huggin gface
Fand delete :d
Fit saves the checkpoint
Fand continue training
PSo is checkpoint 1 completed here?
PIn the screenshot
Fyes saved
Fcheck folder
Pbut why the individual epochs are still in process none is 100% complete
PI am getting confused
Fwhat do you mean?
Pi mean for checkpoint 1 the epochs status bar is not fully filled.does every epoch status need to be filled till 100% ?
Fit shows entire progress
Fit is steps based training actually as you see
Pok ok got it so checkpoint is safe to upload to huggingface and deleted from the instance right?
Pcheckpoint 1
Pok thanks for clearing my doubt doctor
Fsure no issues
P@Furkan Gözükara SECourses is there any way to pause the training for a while and resume again
Pin masscompute
Pwhats the fastest way to upload 24 gb to huggingface ,or is possible to save to my pc local drive ?
Fwe have notebook
FPatreon
Get more from SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts on Patreon
FReddit
Explore this post and more from the SECourses community
TWhat is different in this training?