i wish i knew the science behind how merging works but i just mix them up and hope for the best
i wish i knew the science behind how merging works but i just mix them up and hope for the best
SThis man Sciences
Fthere is no science 
it averages neural network weights and you expect to work
it averages neural network weights and you expect to work
Shahah well i just mean in a basic sense, like if i do 50% + 50% of one model and then merge that with another at 50% does it equal out to be 25% +25% +50% or what lol
Sor am i actually losing any data or does other data just cover it up to where i can bring other data forward by adjusting weights of tokens and what not
Dhm, I seem to be using less VRAM than supposed to
odd
odd
Fmaybe torch 2.5 diff
Fare you using torch 2.5? vrams tested on torch 2.4
AThanks to your guides and configs I fine tuned an SDXL model with pics of my dog and of me using OneTrainer (will post some results later). The thing I'm struggling with is extracting a Lora from the model. I was able to do this with kohya_ss but the Lora comes out at almost 2Gb. I was wondering am I using wrong settings in kohya_ss or should I extract the Lora and then run the reduce Lora option in kohya_ss?
Fwhy you need to reduce its size?
Fyou can reduce by
Fset precision to bf16 or fp16 compare each one quality
Fset rank 128 or lower if you need lower size
Fmay reduce quality
Fi use kohya gui extractor
AYeah, using the Kohya gui. Main reason I'd like to reduce the size is I wanted to share it + my hard drive was getting full (I really should buy a bigger drive). I played a bit with the resize option in the gui. Dropped it to a rank of 64 and the final size was about 380MB without noticeable quality loss. I was mainly curious if there was a recommend way to get smaller Loras.
ABut you are right. I should just follow your example of just testing the quality of the Lora at different ranks and precision till I find a good mix.
Dnot sure, I just installed from the files in the latest kohya pack
DI did install the torch 2.5 one too
NAppreciate you sharing your results, may test extracting mine too
SDr Furkan, i hope evertyting is ok, im traying to make my firt lora model in flux, i have a 4070 super ti 16 gigas vram this speed is good? if i install the torcho 2.5 could increase speed? im use the rank 5 setting
Fspeed is good . after this try torch 2.5 and see if improves. i think may improve
Sthank you Dr Furkan Im using 44 images maybe i will try with fewer the next time
PCan I ask if a 1080 8gb card can train lora? If not, is the 3060 capable?
Fit cant since 1080 doesnt have bf16
Fbut 3060 perfectly can
PThank you for letting me know. With 3060 how long does it take to train on average?
Fit would take like 10 hours to get a perfect quality
Fbecause we have to use huge optimization
Fhowever there may come a new technique
Ftraining a single layer
Fwaiting kohya on that
DSeems like vast majority of character Loras work even without including their supposed trigger words. Interesting
Fflux has internal encoding
Fso every image fully captioned
Dnot sure what you two mean. I have used a character lora with flux recently. trigger word (which was the name of the person) was necessary. using the lora without it already did something, but not everything
Fif you dont use trigger word and if it is not overfit you wont get exactly that person
Fbecuase you cant know exact internal encoding / caption 
SDr Furkan i hope evrything is ok, wht do you think could be the best, cost/benefit amount fo dataset image for training?
Fi plan to test with 100 images
Fcurrently works as low as 15 very good
Dhm, does Lora merging work for flux?
Fthey added some tools but didnt test yet
DI remember results weren't always good previously, especially if one wanted to merge loras of different sizes
MWhen using Dreambooth training, what is the formula for total steps when using multiple GPUs? In the video about Flux training it seemed like you would multiply everything by the number of GPUs, so 100 images, 1 repeat, batch size 1, 10 epochs would be 1000 steps on a 1 GPU, 4000 steps on a x4 GPU machine. However, when running a Dreambooth finetune, the system insists it's going to be 1000 steps unless I actually increase the number of epochs to 10 x the number of GPUs…
Fwhatever you calculate divide to number of gpus
Fso single gpu 1000 steps or 100 epochs becomes 250 steps and 25 epochs with 4 gpus
Falso use number of processes qual to number of gpus
MThat's what I thought, but: I had 175 images, so I set 12 epochs hoping I'd get 8400 steps
MBut the command line logs only showed 1200 steps to go