Software Engineering Courses (SECourses)•2y ago

With an RTX 4070 12gb, training 1024x at float16 using your tier1sdxl 10gb config, do you know a rou

With an RTX 4070 12gb, training 1024x at float16 using your tier1sdxl 10gb config, do you know a rough estimate of how long it would take to train using 25 images? Just a ballpark figure. I'm wondering if I should I set it and leave the house to run errands for a few hours, or if I should set it and leave it to run overnight.

NNAMU With an RTX 4070 12gb, training 1024x at float16 using your tier1sdxl 10gb confi...

Furkan Gözükara SECourses•4/5/24, 9:20 PM

you can calculate

Furkan Gözükara SECourses•4/5/24, 9:20 PM

what is your second / it

NAMUOP•4/5/24, 9:21 PM

Around 3.6 to 3.9

NNAMU Around 3.6 to 3.9

Furkan Gözükara SECourses•4/5/24, 9:46 PM

ok lets say 4 at worst

Furkan Gözükara SECourses•4/5/24, 9:46 PM

so 25 images with lets say 140 epoch

Furkan Gözükara SECourses•4/5/24, 9:46 PM

25 * 140 * 2 = 7000 steps

Furkan Gözükara SECourses•4/5/24, 9:47 PM

so it would be 7000 * 4 / 60 / 60 = 7.8 hours

NAMUOP•4/5/24, 10:35 PM

Overnight it is, then! Thanks for doing the math for me. I can do the math, I just don't enjoy the process, lol.

NAMUOP•4/5/24, 10:36 PM

Although looking at cmd, it looks like I'm averaging around 2s/it - so maybe if I'm lucky, around 4 hours

NAMUOP•4/6/24, 12:19 AM

So I stopped the training at 3100 and tested the saved model, and if I use all the common tags from the training in the same prompt, I get a pretty damned good result. But if I remove all or even just some of those common tags, or if I try to use any kind of creative prompt (like "sitting in a golden throne wearing golden armor"), the face likeness goes away.

Furkan Gözükara SECourses•4/6/24, 1:07 AM

recording onetrainer tutorial already over 2 hours

NNAMU So I stopped the training at 3100 and tested the saved model, and if I use all t...

Furkan Gözükara SECourses•4/6/24, 1:08 AM

did you train with captions?

Furkan Gözükara SECourses•4/6/24, 1:08 AM

you can see caption effect : https://www.patreon.com/posts/compared-effect-101206188

Patreon

Compared Effect Of Image Captioning For SDXL Fine-tuning / DreamBoo...

Get more from SECourses: Tutorials, Guides, Resources, Training, MidJourney, Voice Clone, TTS, ChatGPT, GPT, LLM, Scripts by Furkan Gözükara on Patreon

MMorpheus SX343 I guess the base model only knows DDD extra large boobs 😄 But I dont want these...

Neo•4/6/24, 1:15 AM

Yes, that means the model you're using for inference of that word "boob" is still trained "stronger" than your LoRA concept. If you're training say the concept "girlfriend woman" with captions on your dataset you could leave out any reference to breasts. However it may make your LoRA less flexible. You could also experiment with doing the caption dropout option (available in Kohya) or for an experiment train your LoRA with no captions and see if the accuracy increases...

crapthings•4/6/24, 1:31 AM

hello i have a question
i have 2000 reg images

which should i use？
30 images 1 reg
or
30 images 30 reg

FFurkan Gözükara SECourses you can see caption effect : https://www.patreon.com/posts/compared-effect-10120...

NAMUOP•4/6/24, 1:46 AM

Ahhh, that makes sense! I created the training data before watching the tutorial, so I forgot that the caption text was included in the folder with the training images already. Damn it, back to the drawing board, lol.

NAMUOP•4/6/24, 1:48 AM

@Dr. Furkan Gözükara Someone recomended removing backgrounds for training data images, do you agree with them? I can do that pretty easily.

NAMUOP•4/6/24, 1:49 AM

Or I can use the tool to add the masks

NAMUOP•4/6/24, 1:53 AM

like this

NNAMU @Dr. Furkan Gözükara Someone recomended removing backgrounds for training data ...

EyeSpyBekoAI•4/6/24, 2:11 AM

I can chime in on that a bit. Personally, that gives me disastrous results, but give it a shot if you'd like

NNAMU like this

Furkan Gözükara SECourses•4/6/24, 2:15 AM

yes

Furkan Gözükara SECourses•4/6/24, 3:57 AM

still recording and tutorial is now over 3 hours. this tutorial actually a standalone full course haha

Furkan Gözükara SECourses•4/6/24, 4:12 AM

other than introduction here the tutorial durations :d onetrainer

Furkan Gözükara SECourses•4/6/24, 4:13 AM

i need to record introduction and edit it now

FFurkan Gözükara SECourses still recording and tutorial is now over 3 hours. this tutorial actually a stand...

EyeSpyBekoAI•4/6/24, 4:56 AM

Allah razı olsun hocam!!!

Furkan Gözükara SECourses•4/6/24, 4:58 AM

good day / night

EyeSpyBekoAI•4/6/24, 5:00 AM

Seriously though you’re probably sick of me saying this over and over but I and I’m sure this community is very thankful for everything you do. I’ve learned a ton here.

EyeSpyBekoAI•4/6/24, 5:00 AM

I don’t know when you sleep.

FFurkan Gözükara SECourses good day / night

EyeSpyBekoAI•4/6/24, 5:00 AM

Looks like you do once in a while

NNAMU Ahhh, that makes sense! I created the training data before watching the tutorial...

Alexm•4/6/24, 8:37 AM

do you create samples during training? This can help figure out where it is going wrong. I usually use very specific sampling prompts to see if:

my concept bleeds into other concepts
see if it is flexible on different poses and so on.

best case scenario is that your training does not affect the base models style you are training on and still can do everything the base class can.

if you for example train a specific man, like tom hanks or so, your model should be able to display your concept like it could with your class "man"

so tom hanks in a black suit should look the same as man in a black suit, just with the facial and body features of tom hanks

Alexm•4/6/24, 8:38 AM

thats very difficult to achieve, but once you get a hang of it, you will get good results

Morpheus SX343•4/6/24, 9:03 AM

Any way in Kohya to NOT create checkpoints in the first N steps? Like when I train 5k steps and save 10 checkpoints (that means every 500 steps) - usually the first 3 or 4 checkpoints always look bad, so I'd like to exclude them from saving. So only start with Checkpoint saving from step 2000 onwards
EDIT: I Know I can simply delete the afterwards

EEyeSpyBekoAI Looks like you do once in a while 🤣

Furkan Gözükara SECourses•4/6/24, 10:22 AM

and when i try my wife dont let me sleep :/

MMorpheus SX343 Any way in Kohya to NOT create checkpoints in the first N steps? Like when I tra...

Furkan Gözükara SECourses•4/6/24, 10:27 AM

did you try : Save last N steps
(Optional) Save only the specified number of models (old models will be deleted)

slash•4/6/24, 11:55 AM

hey guys! im having a issue while training a lora . it always gives me 2 faces and i dont know why. my training images are good and even the captioning. but still im getting 2 faces. i trained it on koyhaa. can someone help me, i used majicMIX realistic 麦橘写实 for base model