nevermind. fixed above issues. nnot sure how i fixed the second one but after restart it worked. and
nevermind. fixed above issues. nnot sure how i fixed the second one but after restart it worked. and changing file description limit via terminal.
Finteresting
E@Furkan Gözükara SECourses Hello! I'm using RTX4090 (24Gb)
For full FLUX finetuning I used your config "Rank_1_15500MB_39_Second_IT.json" and It took me 16.5 hours to train 15 images dataset.
Then I tried the config "Quality_1_23100MB_14_12_Second_IT.json" and It took me 23.5 hours to train the same dataset
Why the training goes slower on the second config although it is more vram consuming?
For full FLUX finetuning I used your config "Rank_1_15500MB_39_Second_IT.json" and It took me 16.5 hours to train 15 images dataset.
Then I tried the config "Quality_1_23100MB_14_12_Second_IT.json" and It took me 23.5 hours to train the same dataset
Why the training goes slower on the second config although it is more vram consuming?
Fit means that your initial vram usage was a lot
Fthus you should do this
Fmonitor your shared vram usage
Funtil it doesnt get increase
Fincrease number of block swaps
F1 block swap will reduce vram usage 640 mb for double blocks but reduce speed too
Fso very likely that before starting training you were using at least 800 900 mb vram
FPatreon
Get more from SECourses: Tutorials, Guides, Resources, Training, FLUX, MidJourney, Voice Clone, TTS, ChatGPT, GPT, LLM, Scripts by Furkan Gözü on Patreon
WHi @Furkan Gözükara SECourses is there a way to save the training process into local computer then resume later on mass compute or runpod or something else. Due to training times is very long in case of server restart or crash, are there anyway to save the on going process then resume later ?
Ffor fine tuning yes
Fuse your latest checkpoint
A'block swaps' - I presume that's somewhere in the kohya_ss gui, specifically "double_blocks_to_swap = 5 (in the .toml file)?
AI kept my VRAM right on 23.9 GB using the "Quality_1_23100MB_14_12_Second_IT.json" config (on a 24 GB card). I had to try a few tricks to reduce VRAM usage, such as switching off hardware accelerated GPU scheduling, and selecting 'Adjust for best performance' under Visual Effects in the Control Panel for Advanced System Settings. This helped a little though I still had 400 MB shared on RAM.
Fyou can also compare swap 6 7 speed difference
Fand use fastest one
Fall quality same
AI had tried 4e-06 learning rate, on 26 input images, but all of my epoch results looked similar (I was saving checkpoints every tenth epoch), so I tried again on 5e-06 LR and saw some improvement, though very gradual. I also noticed, right from my earliest checkpoint save, that sample photographic images of a 'woman' looked identical to those for 'ohxw woman', whereas for SDXL dreambooth the two images converged once the model was being over-trained in the new subject.
Fit is true that 05 06 or 03 may work better depending on dataset
Freally hard to set one LR for all
Fi tried to set an LR that will not very undertrain or overtrain
Fbut with my LR as you do more training, you should eventually get your perfect checkpoint
AYes, I eventually got a good result.
WCan you describe more out which step to do it. Where I can load the latest checkpoint and how to calculate the remain step for the latest checkpoint to continue
DI've seen Flux not really converge on realism when the LR is too low, no matter how long you run it. It changes facial features but remains in its plastic AI style with a low LR
Dare the e-06 values above with rank 128 alpha 16?
Fit wont have those
Fyou need to memorize and do yourself
Fdont change alpha we have it affects LR
DI have it at 1 but different LRs, that's why I'm asking. Alpha 16 multiplies the LR by 16 compared to 1
Frank 16 + alpha 16 you trying?
Fyou should make them equal
For your LR need to be researched again
Drank 128 alpha 1
Dthey don't have to be equal
Dalpha is there so you can change the rank without changing the LR
Fthen you need new LR
Dso whatever alpha you use, don't change it. but it doesn't matter what alpha you use.
Factually we make alpha equal to rank
Fso that you can use same LR
Dit's the other way around. if you use alpha 16, you can change the rank without changing the LR. OT (and I think kohya) automatically scale the LR by (alpha / rank)
Fi recently researched this i am 100% sure
Djust keep alpha the same
Fi even talked this with kohya
Falpha is just a multiplier that is used to balance LR for different lora ranks
Dexactly. we mean the same thing 
Ffor example when i moved to single layer, it required 10x more LR
Dkeep your alpha unchanged, keep your LR unchanged, change the rank. it'll work, because LR is automatically scaled.