Nvidia may also make 24 hb rtx 4060 that would be also a good upgrade
Nvidia may also make 24 hb rtx 4060 that would be also a good upgrade
FCan you try half learning rate
NIt would be nice and more affordable for sure
FHalf learning rate can help a lot
F0.000001
NNum batches each epoch = 32
Num Epochs = 100
Batch Size Per Device = 2
Gradient Accumulation steps = 2
Total train batch size (w. parallel, distributed & accumulation) = 4
Text Encoder Epochs: 75
Total optimization steps = 3200
Total training steps = 6400
Resuming from checkpoint: False
First resume epoch: 0
First resume step: 0
Lora: False, Adam: True, Prec: bf16
Gradient Checkpointing: False
EMA: True
UNET: True
Freeze CLIP Normalization Layers: False
LR: 8e-06
Num Epochs = 100
Batch Size Per Device = 2
Gradient Accumulation steps = 2
Total train batch size (w. parallel, distributed & accumulation) = 4
Text Encoder Epochs: 75
Total optimization steps = 3200
Total training steps = 6400
Resuming from checkpoint: False
First resume epoch: 0
First resume step: 0
Lora: False, Adam: True, Prec: bf16
Gradient Checkpointing: False
EMA: True
UNET: True
Freeze CLIP Normalization Layers: False
LR: 8e-06
FYou are using 0.000008
Nok! i am going to try it
FTry half
FIf still try half again 
FI saw your batch size
FAnd gradent
FIt is still experimental to increase learning rate
FSo trying half should help
NBatch Size
1
Gradient Accumulation Steps
1
and learning 0.000001?
1
Gradient Accumulation Steps
1
and learning 0.000001?
C
FYou can test this. Also you can test batch 2 gradient 2 learn 0.4
FBut I first suggest batch 1 gradient 1
FBecause I think implementation could be bugged
Nok
Chmm, @Furkan Gözükara SECourses can you install tokenizer without problems? I think I have to reinstall torch again
FDidn't test recently
Cwont work
FI can test in 30 min
Cdont worry, I got it!
C
C12 images
Cbecause we do not have a big dataset, we dont need files and all files use same 75 tokens
CHere are the class images being generated with the sampe prompt.
Theoretically, if I fine tune with proper weights
Theoretically, if I fine tune with proper weights
Cthe same prompt should in next generation have more red hair
Cand at least, not disturb the images
Cooooh
Fyou are really making experimetns
Fthat others didnt :d
CIs that bad?
C
CIm trying
Fvery low loss looks good
Fcurrently i am focused instruct-pix2pix
CWell I hope I keep you entertained 
Cactually
Cthose are REALLY low numbers
C
Cthats what normally people get
Fyes really low
Flearning super good
C0.12 avg
0.085 avg
0.085 avg
C30% weakerr? Probably less because the upper you have to take into consideration warmup I think?
Fnot sure about that
Fmaybe opposite
Coh wow