What could be the problem, I tried the kohya-ss/sd-scripts branch for lumina 2 (feature-lumina-image) for lumina_train.py. There were a couple of typos img instead of noisy_model_input, but after correcting them, the training started, the loss even goes down, but at the first generation of samples I get noise. If I add "sample_at_first = true" then I see that before training the image is generated normally, but from the very beginning of training there is noise. Now I have
max_timestep = 1000
guidance_scale = 4
timestep_sampling = "nextdit_shift"
discrete_flow_shift = 6.0
model_prediction_type = "raw"
optimizer_args = [ "weight_decay=0.00",]
optimizer_type = "PagedLion8bit"
I tried changing the optimizer, timestep_sampling, discrete_flow_shift, all the same, no results... Already at the 30th step of training there is noise.
for 15 image dataset, were you able to figure what the unet learning rate was ideal?
