default config, 1xA6000 (not spot)
default config, 1xA6000 (not spot)
Dconfig: Batch_Size_7_48GB_GPU_46250MB_28.9_second_it_Tier_1.json
Deven if I change epoch to 1 it throws the error
F<Signals.SIGKILL: 9>. happens with RAM
Factually i tested it on multiple GPU machine
Fmaybe batch size 7 needs more than 48 GB RAM 
Flet me test
DI am currently running normal one, will let me know
Fi am staring a machine now to test
D*I am currently running normal one, will let you know (sorry, typo)
Moh so he wouldn't have knowledge about training? I have a couple of questions about Flux style training can i dm you?
Dnormal (not batch size 7) works
Fi have full flux style training guide did you read?
Fye looks like 48 gb not sufficient
Ffor batch size 7
Fwow i am able to open multiple thinlinc client on windows 11
Fsingle A6000 batch size 7 works
Ffine tuning
F
D@Dr. Furkan Gözükara Update on https://huggingface.co/nyanko7/flux-dev-de-distill New training, catastrophic failure with Apply T5 Attention Mask, and Train T5-XXL enabled. the lora bleeds and is no learning the concepts, the prompt "token class" renders random things like a dog or something on regular flux-dev, on inference using flux-dev-de-distill kinda works sometimes, one image have resembles the next bleeds, all over the place. I going to disable Apply T5 Attention Mask, and Train T5-XXL, one of this options is breaking the model. all tests images have horizontal lines also.
D@Dr. Furkan Gözükara may be the problem wasn't the learning rate, may be is one or both of those Apply T5 Attention Mask, and Train T5-XXL, what do you think?
MI did. Is there much quality difference between using Rank 3 config vs 4x_GPU_Rank_1_SLOW?
MIf not the only think I didn't try is training without captions
Fi think it could be all so every case has to be tested
Fyes small diff
Moh and I didn't try training T5 XXL. If there is no text in my training images would it still improve the training?
DAfter my failure I'm not so sure about "Apply T5 Attention Mask, and Train T5-XXL enabled" you are training the T5, but in inference you load a T5 model that doesn't know your token. forge o SwarmUI load the regular T5 model that was not trained. so I'm not so sure
Moh the t5 model should be trained as well for it to work properly?
D@Dr. Furkan Gözükara may be Apply T5 Attention Mask can help, but I don't understand how can Train T5-XXL helps, because on inference you load a T5 model that was not trained . forge or SwarmUI load the regular T5 model that was not trained
DIs the thing I don't understand, so really IDK now I'm training again with both options disabled, My question can apply to everyone, but in my case is different because I'm training flux-dev-de-distill so in my case is broking something. I'm having artifacts on the images with this on and also is not learning the concepts correctly
D@Dr. Furkan Gözükara Update on https://huggingface.co/nyanko7/flux-dev-de-distill I started new training - "T5 Attention Mask and T5-XXL both disabled" same lr 30 epochs now. I tested the lora checkpoint and all problems are fixed, is going great using regular flux-dev-fp8 for inference, I'm training tree people same class, same prompt changing just the name renders the correct subject, is undertrained but goes very well so far
A
Adid i do something wrong? why is there 17.6 million steps ??
FT5 XXL made minor diff that i didnt notice :d
Fyep you did indeed
Fsub folder
Fmore than 1 repeat
Aah it was at 40
Fyep
A
Ais that too many still
A2206 images
Fsub folders
VThe config could use an update as he made the same mistake as me.
Fi tested config today works as expected
VThe repeat was at 40 when I used it, is that not saved in the config?
Fit is read from folder
Fyou need to fix folder name
Fi explained this in details in tutorial
