Software Engineering Courses (SECourses)•3y ago

@Dr. Furkan Gözükara any clue what part of the pipeline is causing my sdxl dreambooth fine tunes to

@Dr. Furkan Gözükara any clue what part of the pipeline is causing my sdxl dreambooth fine tunes to output noise?

8ba7af1511ab6f3c9e884e3c1d5b01db7c505ac19aeb2d5121668890eea2d030_image.png

stillmotionOP•10/11/23, 3:30 PM

I had it working end to end a few days ago, but I think I broke something. Trying to figure out if the training was bad or if it's an inference problem...

stillmotionOP•10/11/23, 3:31 PM

My hunch is switching to the fp16 vae. Have you encountered similar noise?

Sstillmotion @Dr. Furkan Gözükara any clue what part of the pipeline is causing my sdxl dream...

Furkan Gözükara SECourses•10/11/23, 3:33 PM

VAE

Furkan Gözükara SECourses•10/11/23, 3:33 PM

overcooked unet

stillmotionOP•10/11/23, 3:33 PM

ok, I'll check the earlier checkpoints to see if it's overcooked

stillmotionOP•10/11/23, 3:55 PM

@Dr. Furkan Gözükara ok getting the same output for all the checkpoints, so definitely a vae issue. How can I identify if it's an issue with the inference configuration vs. the vae I configured for training?

stillmotionOP•10/11/23, 3:56 PM

Not convinced I'm loading the vae or unet with the correct dtype

Sstillmotion @Dr. Furkan Gözükara ok getting the same output for all the checkpoints, so defi...

Furkan Gözükara SECourses•10/11/23, 4:46 PM

which vae do you chose during inference?

stillmotionOP•10/11/23, 4:53 PM

The one you linked me last week. This is how I'm loading via diffusers:

        vae = AutoencoderKL.from_pretrained(
            "stabilityai/sdxl-vae", torch_dtype=torch.float16
        )

        vae = AutoencoderKL.from_pretrained(
            "stabilityai/sdxl-vae", torch_dtype=torch.float16
        )

stillmotionOP•10/11/23, 4:54 PM

afaik that one doesn't have the NaN issues

stillmotionOP•10/11/23, 4:56 PM

Ah, I'm not generating samples during training. I'll do that to check if there's an issue with the training, since it uses the same vae and config

stillmotionOP•10/11/23, 5:44 PM

@Dr. Furkan Gözükara ok yeah, my sample looks like this. Probably the way I'm loading the vae

akellio•10/11/23, 6:48 PM

Hi, i have this error now when i run the kohya interface

Zono50•10/11/23, 7:01 PM

What's the best way to fix the blurryness in this photo, upscaling doesn't seem to do anything

Sstillmotion @Dr. Furkan Gözükara ok yeah, my sample looks like this. Probably the way I'm lo...

Furkan Gözükara SECourses•10/11/23, 7:31 PM

what do you use to laod vae?

Furkan Gözükara SECourses•10/11/23, 7:31 PM

you dont use auto1111?,

Aakellio Hi, i have this error now when i run the kohya interface

Furkan Gözükara SECourses•10/11/23, 7:31 PM

use my auto installer

Furkan Gözükara SECourses•10/11/23, 7:31 PM

https://www.patreon.com/posts/for-runpod-kohya-84898806

ZZono50 What's the best way to fix the blurryness in this photo, upscaling doesn't seem ...

Furkan Gözükara SECourses•10/11/23, 7:32 PM

high res fix usually

FFurkan Gözükara SECourses use my auto installer

akellio•10/11/23, 7:32 PM

it's a kaggle problem

Aakellio it's a kaggle problem

Furkan Gözükara SECourses•10/11/23, 7:32 PM

i updted kaggle notebook

Furkan Gözükara SECourses•10/11/23, 7:33 PM

now works better

Furkan Gözükara SECourses•10/11/23, 7:33 PM

we use working directory

Furkan Gözükara SECourses•10/11/23, 7:33 PM

temp is too slow

Furkan Gözükara SECourses•10/11/23, 7:33 PM

fails

akellio•10/11/23, 7:33 PM

the same for the patreon?

Aakellio the same for the patreon?

Furkan Gözükara SECourses•10/11/23, 7:38 PM

yes on patreon

Furkan Gözükara SECourses•10/11/23, 7:38 PM

are you using latest version?

FFurkan Gözükara SECourses you dont use auto1111?,

stillmotionOP•10/11/23, 7:39 PM

I am using sd-scripts directly for training and then using diffusers for inference.

I’ve narrowed down the issue to bf16 mixed precision setting not loading the vae as half precision.

stillmotionOP•10/11/23, 7:40 PM

Not clear to me why the vae loading doesn’t support bf16 and instead loads the full float version

Sstillmotion I am using sd-scripts directly for training and then using diffusers for inferen...

Furkan Gözükara SECourses•10/11/23, 7:41 PM

diffusers inference problematic

Furkan Gözükara SECourses•10/11/23, 7:41 PM

i am giving consultation to 1 company

Furkan Gözükara SECourses•10/11/23, 7:41 PM

they failed to replicate automatic1111 yet

stillmotionOP•10/11/23, 7:41 PM

Possibly, but seems like the issue is with the training at the moment

stillmotionOP•10/11/23, 7:42 PM