I thought it would be faster to train
I thought it would be faster to train
Fnope
Aalways train on fp32
Athen inference on fp16<=
Athe logic is to save time on inference not on training for the best quality
LGood morning, gentlemen, i've been using Tier1_48_GB_Faster_v2.json for dreambooth on runpod and constantly getting the training terminated on text encoders training part
So,
subprocess.CalledProcessError: Command '['/opt/environments/python/kohya/bin/python3.10', '/workspace/kohya_ss/sd-scripts/sdxl_train.py', '--config_file', '/workspace/kohya_ss/outputs/config_dreambooth-20250719-133856.toml', '--max_grad_norm=0.0', '--no_half_vae', '--train_text_encoder', , '--learning_rate_te2=0']' returned non-zero exit status 2.
As i understand kohya wants non-zero parameter for text_encoder2
should i insert the same as on te1, 0.000003?
So,
subprocess.CalledProcessError: Command '['/opt/environments/python/kohya/bin/python3.10', '/workspace/kohya_ss/sd-scripts/sdxl_train.py', '--config_file', '/workspace/kohya_ss/outputs/config_dreambooth-20250719-133856.toml', '--max_grad_norm=0.0', '--no_half_vae', '--train_text_encoder', , '--learning_rate_te2=0']' returned non-zero exit status 2.
As i understand kohya wants non-zero parameter for text_encoder2
should i insert the same as on te1, 0.000003?
Fhi what is the exact message?
Fit could be out of RAM
LHey guys. I'm currently training a flux style finetune using kohya with a dataset consisting of around 1800 images total. When leaving the max epoch to 0 (default) it was automatically set to 6. This seemed strange to me, I've never used a dataset this large. I'm using the "Batch_Size_7_48GB_GPU_46250MB_29.1_second_it_Tier_1" preset as provided by Dr. Furkan. Should I tweak another variable like learning rate for a dataset this big? I set max epochs to 10 and it says it will train for 20 hours on an A6000 GPU. I thought I'd ask before I have it train for 20 hours for it to end up being trash.
LCanceled it for now just in case someone more experienced tells me I've made a fatal error.
Fare these 1800 images are consistent?
Ffor 1800 images i recommend train up to 15 epoch , take a checkpoint every epoch and compare later
LThanks for the response! They are high quality screenshots from a show with a unique art style, consistent in style only. All of them are uniquely captioned in the same format to describe the image, with a triggerword for the style at the start of the caption. I'll try as you suggested. One more question: Can I train 10 epochs, and with the resulting model, continue training for another 5 epochs in a separate instance (using the resulting model as a base) to get the same or similar result as if I were training 15 epochs from the start? A checkpoint per epoch would require me to regularly offload the models onto another server due to disk space, risking messing this up and epochs not saving.
LAlso, although maybe not particularly relevant. They are actually 2 sets of the same 900 images in different aspect ratios, according to my own testing and other testing I've seen, training on different aspect ratios improves results. Particularly with style finetunes. Never tried it with this many images though.
LThat was exactly it.
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.20 GiB of which 16.62 MiB is free. Process 2510593 has 79.18 GiB memory in use.,
training was successful on batch size=2 on h100 sxm
This thing is demanding indeed, consumed like 77gb vram
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.20 GiB of which 16.62 MiB is free. Process 2510593 has 79.18 GiB memory in use.,
training was successful on batch size=2 on h100 sxm
This thing is demanding indeed, consumed like 77gb vram
Fwith our config 10+5 = 15 from 0
Fwell it should work decent
Fmake sure to enable bucketing
Fyes batch size 2 for dreambooth requires huge vram and ram
Fbut lora works great
Awhat config should i use on H100 @Furkan Gözükara SECourses batch size 7 48gb or 48gb Q1? whats the best quality?
LIs 20 images enough for dreamboothoing? or i need more ( i have 300)
Fbut if you want speed use batch size 7
Ano i want quality
Fmore better
Aso tier 1 ok thanks
Fdefinitely
Fyes
Lrepeats the same? 150-200 or i need to lower?
Abtw doctor for multitalk very good and stable results gives the default t2v lighxv2 rank 32
Flower
Ffor 300 images do like maximum 100 epoch and compare
Fwell i didnt test to many times
Aim closing my h100 soon
Aget ready to test
Athanks
Fso your h100 failing at sage attention?
For what error you getting
Ano h100 good but c10 error
Aon first launch
A
U@Furkan Gözükara SECourses Hello,
I was wondering if you have a python code that I can refer/modify for finetuning instead of using the kohya gui? I want to load the parameters and train without having to open the interface and load the parameters all the time.
I was wondering if you have a python code that I can refer/modify for finetuning instead of using the kohya gui? I want to load the parameters and train without having to open the interface and load the parameters all the time.
Fgui runs cmd code literally
Fpay attention to cmd
UYeah, I tried it, but the results I get vary. However, from the GUI, I get good results. I’ll check if the code uses the correct parameters again.
Fye kohya is based on cmd command
Fso you need to make it accurate
UI was able to get it working.
Also kohya does not support chroma finetuning do you know any other open source model that supports chroma finetuning?
Also kohya does not support chroma finetuning do you know any other open source model that supports chroma finetuning?
FGitHub
The ultimate training toolkit for finetuning diffusion models - ostris/ai-toolkit