Succesfully trained a model with clip l text encoder, results are better now and so are prompting
Succesfully trained a model with clip l text encoder, results are better now and so are prompting
Fvram usage shouldnt be changed
Fyou used same LR?
Vyou can use the fp8 t5
Vuse half learning rate for text encoder
Vor else you will cook your model
Fthis shouldnt make difference
Fi think it auto cast
Fand we train only clip L
Vim getting less vram than before so im not sure whats changed
Flet me check vram too
Fi will still record video waiting clip l training results would delay further 
Vit may be this
VIn FLUX.1 LoRA training, when
--fp8_basefloat8_e4m3fnflux_minimal_inference.pyfp8 (float8_e4m3fn)--flux_dtypeye it just let you load fp8
Ffp16 was still casted to fp8
Vi used this here --fp8_base_unet
Vand clip is trained at bf16
Vim using sd scripts directly
Fkk
Fon windows or linux?
Vlinux
Vdont bother with windows
Fok. if i can lower LR + clip L
Fit can lesser overfit
V7e-05 for unet
Vjust use higher weight decay for overfitting
Fi already use lower than that
Vyou're using adfactor though
Vadamw8bit i get better results
Fye i use most optimized VRAM to cover more GPUs
Fi got a new finding
Fshift value 1
Vi thought it was common to use shift value 1
Dstarting a training with TE
Vi've been using it this whole time
F
Fi was using default value 3
Falso this is shift value sampling
Dlet's see if still bleeds......
Fyou mean discrete_flow_shift right?
F
Fcolors better
Farmor better too
VOverfitted still
F
Falthough likeliness a bit reduced
Druns slower on WSL with TE
Mdid you upgrade to torch 2.4.0?
Dyes I did 2 days ago, I was having 3.5 to 4 it/s
