it could be out of RAM
it could be out of RAM
LHey guys. I'm currently training a flux style finetune using kohya with a dataset consisting of around 1800 images total. When leaving the max epoch to 0 (default) it was automatically set to 6. This seemed strange to me, I've never used a dataset this large. I'm using the "Batch_Size_7_48GB_GPU_46250MB_29.1_second_it_Tier_1" preset as provided by Dr. Furkan. Should I tweak another variable like learning rate for a dataset this big? I set max epochs to 10 and it says it will train for 20 hours on an A6000 GPU. I thought I'd ask before I have it train for 20 hours for it to end up being trash.
LCanceled it for now just in case someone more experienced tells me I've made a fatal error.
Fare these 1800 images are consistent?
Ffor 1800 images i recommend train up to 15 epoch , take a checkpoint every epoch and compare later
LThanks for the response! They are high quality screenshots from a show with a unique art style, consistent in style only. All of them are uniquely captioned in the same format to describe the image, with a triggerword for the style at the start of the caption. I'll try as you suggested. One more question: Can I train 10 epochs, and with the resulting model, continue training for another 5 epochs in a separate instance (using the resulting model as a base) to get the same or similar result as if I were training 15 epochs from the start? A checkpoint per epoch would require me to regularly offload the models onto another server due to disk space, risking messing this up and epochs not saving.
LAlso, although maybe not particularly relevant. They are actually 2 sets of the same 900 images in different aspect ratios, according to my own testing and other testing I've seen, training on different aspect ratios improves results. Particularly with style finetunes. Never tried it with this many images though.
LThat was exactly it.
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.20 GiB of which 16.62 MiB is free. Process 2510593 has 79.18 GiB memory in use.,
training was successful on batch size=2 on h100 sxm
This thing is demanding indeed, consumed like 77gb vram
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.20 GiB of which 16.62 MiB is free. Process 2510593 has 79.18 GiB memory in use.,
training was successful on batch size=2 on h100 sxm
This thing is demanding indeed, consumed like 77gb vram
Fwith our config 10+5 = 15 from 0
Fwell it should work decent
Fmake sure to enable bucketing
Fyes batch size 2 for dreambooth requires huge vram and ram
Fbut lora works great
Awhat config should i use on H100 @Furkan Gözükara SECourses batch size 7 48gb or 48gb Q1? whats the best quality?
LIs 20 images enough for dreamboothoing? or i need more ( i have 300)
Fbut if you want speed use batch size 7
Ano i want quality
Fmore better
Aso tier 1 ok thanks
Fdefinitely
Fyes
Lrepeats the same? 150-200 or i need to lower?
Abtw doctor for multitalk very good and stable results gives the default t2v lighxv2 rank 32
Flower
Ffor 300 images do like maximum 100 epoch and compare
Fwell i didnt test to many times
Aim closing my h100 soon
Aget ready to test
Athanks
Fso your h100 failing at sage attention?
For what error you getting
Ano h100 good but c10 error
Aon first launch
A
U@Furkan Gözükara SECourses Hello,
I was wondering if you have a python code that I can refer/modify for finetuning instead of using the kohya gui? I want to load the parameters and train without having to open the interface and load the parameters all the time.
I was wondering if you have a python code that I can refer/modify for finetuning instead of using the kohya gui? I want to load the parameters and train without having to open the interface and load the parameters all the time.
Fgui runs cmd code literally
Fpay attention to cmd
UYeah, I tried it, but the results I get vary. However, from the GUI, I get good results. I’ll check if the code uses the correct parameters again.
Fye kohya is based on cmd command
Fso you need to make it accurate
UI was able to get it working.
Also kohya does not support chroma finetuning do you know any other open source model that supports chroma finetuning?
Also kohya does not support chroma finetuning do you know any other open source model that supports chroma finetuning?
FGitHub
The ultimate training toolkit for finetuning diffusion models - ostris/ai-toolkit
Fi am not sure though
Falso make a reply here
FGitHub
With the FLUX.1-schnell based Chroma model gaining popularity is there any chance of it being added, for lora trainng ?
UI tried ai tool kit but I could only find lora training. I will check again.
U"I’d like to support this request. It would be beneficial if the developers could add support for the Chroma model.
Chroma is an 8.9B parameter model based on FLUX.1-schnell. More information can be found here: huggingface.co/lodestones/Chroma
Additionally, ai-toolkit and the diffusion-pipe project at github.com/tdrussell/diffusion-pipe already supports Chroma. (LoRA/fine-tuning).
Thank you for the fantastic sd-scripts project; your work is greatly appreciated!"
I came across this comment by blackmagix24. I will give diffusion pipe a shot. if you happen to test it in the future, I would love to see a video from you for chroma fine tuning along with a comparison with flux dev
Chroma is an 8.9B parameter model based on FLUX.1-schnell. More information can be found here: huggingface.co/lodestones/Chroma
Additionally, ai-toolkit and the diffusion-pipe project at github.com/tdrussell/diffusion-pipe already supports Chroma. (LoRA/fine-tuning).
Thank you for the fantastic sd-scripts project; your work is greatly appreciated!"
I came across this comment by blackmagix24. I will give diffusion pipe a shot. if you happen to test it in the future, I would love to see a video from you for chroma fine tuning along with a comparison with flux dev
Fit is just weaker
Fwhat makes flux different is that it is literally built to make money