are you able to share the results for learning purposes?
are you able to share the results for learning purposes?
Ualso for batch size, what number do you recommend? I'm renting A100 gpu. I'm testing with a batch size of 1 for 15 images. I want to increase it but I'm afraid the model might overfit. I can test them myself but I'm paying by the hour so testing all of them is going to burn me out 
Fi recommend bs 1
Fi planned the test but later didnt have time 
UThank you very much. Your testing evaluation is saving my time 
Uone more question
you used constant LR scheduler, why not cosine or cosine with restarts?
With the constant scheduler, the model doesn't learn. The training loss doesn't decrease. Is this okay?
you used constant LR scheduler, why not cosine or cosine with restarts?
With the constant scheduler, the model doesn't learn. The training loss doesn't decrease. Is this okay?
Ftraining loss useless
Fconstant also gives you chance to continue perfectly fine
KWhat could be the problem, I tried the kohya-ss/sd-scripts branch for lumina 2 (feature-lumina-image) for lumina_train.py. There were a couple of typos img instead of noisy_model_input, but after correcting them, the training started, the loss even goes down, but at the first generation of samples I get noise. If I add "sample_at_first = true" then I see that before training the image is generated normally, but from the very beginning of training there is noise. Now I have
max_timestep = 1000
guidance_scale = 4
timestep_sampling = "nextdit_shift"
discrete_flow_shift = 6.0
model_prediction_type = "raw"
optimizer_args = [ "weight_decay=0.00",]
optimizer_type = "PagedLion8bit"
I tried changing the optimizer, timestep_sampling, discrete_flow_shift, all the same, no results... Already at the 30th step of training there is noise.
max_timestep = 1000
guidance_scale = 4
timestep_sampling = "nextdit_shift"
discrete_flow_shift = 6.0
model_prediction_type = "raw"
optimizer_args = [ "weight_decay=0.00",]
optimizer_type = "PagedLion8bit"
I tried changing the optimizer, timestep_sampling, discrete_flow_shift, all the same, no results... Already at the 30th step of training there is noise.
Fnever tried lumina 2
K@Furkan Gözükara SECourses is theres a way to understand how to improve speed? i followed all the steps from the videos on and rtx 5090
Fwhich config?
Ktest_24GB_GPU_22900MB_7.3_second_it_Tier_1.json
Fuse 48 gb config
Frtx 5090 can do it since full model uses like 28.5 gb
Kok thanks
Fyou are welcome i got to sleep
Ki changed to 48 but stills runs at the same speed.
ae = "X:/AI/Kohya/ae.safetensors" do you see anything wrong with the toml?
I really appreciate your help
blocks_to_swap = 0
bucket_no_upscale = true
bucket_reso_steps = 64
cache_latents = true
cache_latents_to_disk = true
cache_text_encoder_outputs = true
cache_text_encoder_outputs_to_disk = true
caption_dropout_every_n_epochs = 0
caption_dropout_rate = 0
ae = "X:/AI/Kohya/ae.safetensors" do you see anything wrong with the toml?
I really appreciate your help
blocks_to_swap = 0
bucket_no_upscale = true
bucket_reso_steps = 64
cache_latents = true
cache_latents_to_disk = true
cache_text_encoder_outputs = true
cache_text_encoder_outputs_to_disk = true
caption_dropout_every_n_epochs = 0
caption_dropout_rate = 0
Ksingle_blocks_to_swap = 0
t5xxl = "X:/AI/Kohya/t5xxl_fp16.safetensors"
t5xxl_max_token_length = 512
timestep_sampling = "sigmoid"
train_batch_size = 1
train_blocks = "all"
train_data_dir = "X:\AI\Kohya\SwarmUI_Model_Downloader_v54\SwarmUI\Models\diffusion_models\img"
vae_batch_size = 4
wandb_run_name = "miakillamodel"
xformers = true
t5xxl = "X:/AI/Kohya/t5xxl_fp16.safetensors"
t5xxl_max_token_length = 512
timestep_sampling = "sigmoid"
train_batch_size = 1
train_blocks = "all"
train_data_dir = "X:\AI\Kohya\SwarmUI_Model_Downloader_v54\SwarmUI\Models\diffusion_models\img"
vae_batch_size = 4
wandb_run_name = "miakillamodel"
xformers = true
Fwait at least 100 steps
Falso show me nvitop output
K
Fyes looking accurate
Fwhat is speed?
K4.22
Fyes a little bit slow
Fi dont know what causes exactly but kohya working slower than before
Fhe made something worse
Fwe cant fully utilize gpu for some reason
F
Kok, at least i know Im on the right path, thanks so much
Fyou are welcome
F10 epoch is too low
Fhow many images you have in dataset?
Kyeah i paused just to test, i have 171 faces with same hairstyle different emotions, 53 body different poses, a bit of different hair styles
Fwhat you want to teach?
Ka person, trying to keep the same hairstyle, but also the option to change the hairstyle with prompts
Fok it is easy. 171 images are all from same person right?
Fa bit of different hair styles are ok
Fso what you should do is
Kyeah mostly same hair cut, the ones with full body have different haircuts
Fjust train with ohwx woman
Fthen when prompting
Fdescribe the hair style in details
Fand it should work 95%+
Kok, ill keep training from epoch 10 forward, and ill try that in a bit, thanks so much
Fyou are welcome
Fi trained with 256 images
Fand 70 epoch was best
Fand it is excellent quality
Kexcelent, yeah thats what ill try to do
