repeats are a kohya thing, 1 epoch in the general context of ML usually means going once through your dataset. And look here, 2,5 minutes. Even if you used multiple GPUs, it cannot mean much more than going through your dataset once with the s/it I have seen so far on flux

dxqbOP•9/14/24, 12:55 PM

that being said, I share your doubts

Ddxqb repeats are a kohya thing, 1 epoch in the general context of ML usually means go...

Furkan Gözükara SECourses•9/14/24, 1:31 PM

i know but civitai already uses kohya repeating

when i first trained they were at least

Ddxqb repeats are a kohya thing, 1 epoch in the general context of ML usually means go...

Furkan Gözükara SECourses•9/14/24, 1:32 PM

their description i think means that you dont get checkpoints not necessarly 1 repeat

Yog-Sototh•9/14/24, 2:41 PM

I have question about training and captions please : How many images you think, for a Pose lora including 2 or multiple characters should be in the folder, and repeats ? And also my captionning, can I send one here with the related image ? It is Sfw but may contain violence, soo i prefer to ask ^^

YYog-Sototh I have question about training and captions please : How many images you think, ...

Furkan Gözükara SECourses•9/14/24, 2:44 PM

you can dm me

novsky•9/14/24, 4:04 PM

sooo...I've tried some lora trainings recently. What can I say. Sometimes results are pretty good on 70-80 epochs (once on 50 epochs) and sometimes 200 epochs are not good enough. I guess it all depends on dataset.

novsky•9/14/24, 4:06 PM

and training on realistic dataset will not give you any cartoonish (comics) style output. Because it's too overtrained on realistic images. I did sample check every 40 epochs (5 checks during training) and after 80th epoch cartoon style becomes realistic

wardensc2•9/15/24, 10:38 AM

Hi @Furkan Gözükara SECourses can you create another train preset using 24gb Vram to train fp16 Lora. I trained using Rank_3_18950MB. The lora is fp8 compare with fp16 training in your Rank 1 or Rank 2 presets, the prompt understanding of the fp8 lora is reduce and the result of fp8 is so far not satisfy my need

wardensc2•9/15/24, 10:38 AM

Thank you in advance

wardensc2•9/15/24, 10:39 AM

Maybe you can create new preset taking longer time to train but still use only 24gb Vram

dxqbOP•9/15/24, 10:51 AM

I have just started with Flux, but using onetrainer and bf16 I am below 12 GB vram...

dxqbOP•9/15/24, 10:51 AM

or do you mean the base model in fp16, not the Lora weights?

dxqbOP•9/15/24, 11:02 AM

if so, I doubt that makes a difference. even during inference there is not much different between fp8 and NF4

Wwardensc2 Hi <@205854764540362752> can you create another train preset using 24gb Vram to ...

Furkan Gözükara SECourses•9/15/24, 11:36 AM

for fp16 you have to reduce resolution

Furkan Gözükara SECourses•9/15/24, 11:36 AM

no other way

Ddxqb I have just started with Flux, but using onetrainer and bf16 I am below 12 GB vr...

Furkan Gözükara SECourses•9/15/24, 11:37 AM

he means training

dxqbOP•9/15/24, 11:37 AM

? I also mean training

dxqbOP•9/15/24, 11:38 AM

I'm just saying that even during inference a low-precision base model doesn't make a difference, so how would it during training. The Lora weights are still bf16+

dxqbOP•9/15/24, 11:38 AM

fascinating how Flux learns without any distortion. it just slowly adjusts

Ddxqb I'm just saying that even during inference a low-precision base model doesn't ma...

Furkan Gözükara SECourses•9/15/24, 11:42 AM

it makes some

Furkan Gözükara SECourses•9/15/24, 11:42 AM

not very dramatic

dxqbOP•9/15/24, 5:50 PM

I'm getting 6s/it at home on a 4070, but 14 s/it on an A6000. Do you have any idea why this could be? identical config

dxqbOP•9/15/24, 5:50 PM

flux lora

Ddxqb I'm getting 6s/it at home on a 4070, but 14 s/it on an A6000. Do you have any id...

Furkan Gözükara SECourses•9/15/24, 6:04 PM

interesting

Ddxqb I'm just saying that even during inference a low-precision base model doesn't ma...

wardensc2•9/16/24, 12:09 AM

No, base model fp8 which uses to train lora does affect the lora. For example 2 train 2 lora same dataset but one with fp16 based model one with fp8. The lora with fp16 have better prompt understand than fp8. My training is person lora. While the person quality is still good with both loras. The prompt understanding every subjects outside of the person is reduce with fp8 lora. For example: my prompt have something like little angels playing in the background. The fp16 lora show some angel with body of human and wing but the fp8 showing some birds.

FFurkan Gözükara SECourses interesting

dxqbOP•9/16/24, 6:43 AM

was a mistake, images were cached at a different resolution on the 4070

Ddxqb was a mistake, images were cached at a different resolution on the 4070

Furkan Gözükara SECourses•9/16/24, 10:45 AM

this is actually one of my fears

Furkan Gözükara SECourses•9/16/24, 10:45 AM

different folders should be used for different resolutions

Furkan Gözükara SECourses•9/16/24, 10:45 AM

kohya may fail to properly cache

jkgnl•9/16/24, 11:18 PM

I am training currently a Lora with Flux using 20 photos and 200 Epoch, is it normal it will take 4 tot 5 hours using an RTX 4090 locally? Or did I miss any optimization setting or script?

Jjkgnl I am training currently a Lora with Flux using 20 photos and 200 Epoch, is it no...

Furkan Gözükara SECourses•9/16/24, 11:25 PM

sounds accurate

Furkan Gözükara SECourses•9/16/24, 11:25 PM

what is your step speed?

Furkan Gözükara SECourses•9/16/24, 11:25 PM

which rank config?

jkgnl•9/16/24, 11:38 PM

Step speed is 4.71s/it , how can I see my rank config?

jkgnl•9/16/24, 11:39 PM

I used a Saas solution where I trained on the same 20 photos using Flux Lora, it took me only 30 minutes, any idea what can cause the difference in speed?

Jjkgnl I used a Saas solution where I trained on the same 20 photos using Flux Lora, it...

Furkan Gözükara SECourses•9/16/24, 11:49 PM

from zip file which one you loaded?_

Jjkgnl I used a Saas solution where I trained on the same 20 photos using Flux Lora, it...

Furkan Gözükara SECourses•9/16/24, 11:49 PM

they are lower quality simple as that

Furkan Gözükara SECourses•9/16/24, 11:49 PM

unless expensive h100 one

what are they offering here?

Similar Threads

Similar Threads

Similar Threads