Software Engineering Courses (SECourses)•3y ago

Agree I made some test for speed/memory. No gradient checkpointing: xformers + Mem Efficient Attenti

Agree I made some test for speed/memory. No gradient checkpointing:
xformers + Mem Efficient Attention = xformers without Mem. E. attention
Same speed, same memory usage.

mikemenders•10/9/23, 6:37 AM

I noticed a very interesting thing. I'm training for style and I haven't changed anything else, just the number of epochs from 10 to 30, and I get completely different sample images and the training values are absolutely different. I'm using Prodigy now.

PPraveen Miriyala I'm training LoRa SDXL on my 3060 with 12GB VRAM, 768*768 resolution and it's sh...

Furkan Gözükara SECourses•10/9/23, 9:27 AM

yep looking decent

Mmikemenders It's interesting, because I also have an RTX 3060 12 GB card and I can train a p...

Furkan Gözükara SECourses•10/9/23, 9:27 AM

512x512 much lower resolution

JJM Is 6700 steps enough for sdxl?

Furkan Gözükara SECourses•10/9/23, 9:28 AM

it is decent steps yes

MMetaphysix Agree I made some test for speed/memory. No gradient checkpointing: xformers + M...

Furkan Gözükara SECourses•10/9/23, 9:28 AM

yep i noticed same

FFurkan Gözükara SECourses yep looking decent

Praveen Miriyala•10/9/23, 9:28 AM

but result was not good

Mmikemenders I noticed a very interesting thing. I'm training for style and I haven't changed...

Furkan Gözükara SECourses•10/9/23, 9:28 AM

i never got good results with Prodigy

PPraveen Miriyala but result was not good ☹️

Furkan Gözükara SECourses•10/9/23, 9:29 AM

try bf16 too

Furkan Gözükara SECourses•10/9/23, 9:29 AM

but reason could be 768x768

Furkan Gözükara SECourses•10/9/23, 9:29 AM

also add this parameter to optimizer : --max_grad_norm=0.0

Furkan Gözükara SECourses•10/9/23, 9:29 AM

i will do another research for LoRA

Praveen Miriyala•10/9/23, 9:30 AM

ok i will follow these tips for next training

JM•10/9/23, 9:38 AM

: A tensor with all NaNs was produced in VAE. This could
be because there's not enough precision to represent the picture. Try adding --no-half-vae commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check. How to fix?

SuperTurboHero•10/9/23, 10:05 AM

I wonder what is quality difference if I don't use classification images? training would be much faster, but would I lose a lot of quality?

mikemenders•10/9/23, 10:40 AM

I used this parameters for Prodigy, and trained SD 1.5 base model: "use_bias_correction=True" "weight_decay=0.5" "decouple=True" "betas=(0.9, 0.99)" "d_coef=2" "safeguard_warmup=False"

JJM : A tensor with all NaNs was produced in VAE. This could be because there's not ...

Furkan Gözükara SECourses•10/9/23, 10:41 AM

this happens when model totally overtrained

FFurkan Gözükara SECourses this happens when model totally overtrained

JM•10/9/23, 10:57 AM

I don't use with model lora I use model checkpoint Every model doesn't work.Basicly sd 1.5 I found this problem.

Sandstormleader•10/9/23, 10:57 AM

so is SDXL training essentially the same as 1.5 training just using an SDXL model? @Dr. Furkan Gözükara

SSandstormleader so is SDXL training essentially the same as 1.5 training just using an SDXL mode...

Furkan Gözükara SECourses•10/9/23, 11:15 AM

the parameters changes too

Furkan Gözükara SECourses•10/9/23, 11:15 AM

learning rate

Furkan Gözükara SECourses•10/9/23, 11:15 AM

Sandstormleader•10/9/23, 11:18 AM

so not 0.000001

SSandstormleader so not 0.000001

Furkan Gözükara SECourses•10/9/23, 11:19 AM

yes

Furkan Gözükara SECourses•10/9/23, 11:19 AM

I am searching best parameters for SDXL DreamBooth for weeks now

Furkan Gözükara SECourses•10/9/23, 11:38 AM

Huge Stable Diffusion XL (SDXL) Text Encoder DreamBooth training comparison

U-NET is always trained

PNG info (prompts) are in alts

All images are 1024x1024 so download full sizes

https://twitter.com/GozukaraFurkan/status/1711345323900100643

Benjamin•10/9/23, 11:40 AM

v23.10.0 adetailer got released. Just noticed because now in the preview of the image a red box appears around the detected face

FFurkan Gözükara SECourses Huge Stable Diffusion XL (SDXL) Text Encoder DreamBooth training comparison U-N...

Benjamin•10/9/23, 11:41 AM

good job, from a first glance it does not look like it is changing that much? what is your impression

BBenjamin v23.10.0 adetailer got released. Just noticed because now in the preview of the ...

Furkan Gözükara SECourses•10/9/23, 12:44 PM

2023-10-07

- v23.10.0
- 허깅페이스 모델을 다운로드 실패했을 때, 계속 다운로드를 시도하지 않음
- img2img에서 img2img단계를 건너뛰는 기능 추가
- live preview에서 감지 단계를 보여줌 (PR #352)

BBenjamin good job, from a first glance it does not look like it is changing that much? wh...

Furkan Gözükara SECourses•10/9/23, 12:45 PM

i think it still adds some more extra details

Furkan Gözükara SECourses•10/9/23, 12:45 PM

so 2e-6 looks like nice. 3e-6 also nice but a little bit starting to cook

FFurkan Gözükara SECourses i think it still adds some more extra details

Benjamin•10/9/23, 12:47 PM

how did it influence the training time?

Benjamin•10/9/23, 12:48 PM

ah you already wrote in patreon it is slower

BBenjamin ah you already wrote in patreon it is slower

Furkan Gözükara SECourses•10/9/23, 12:54 PM

ye a little bit more slower

Furkan Gözükara SECourses•10/9/23, 12:54 PM

like 1.5 second / it to 1.75 second / it

Stefanoid9343•10/9/23, 1:11 PM

Thank you so much to @Dr. Furkan Gözükara and @Sandstormleader after several hours of training and solving some errors the model came as I expected this is definitely a game changer process

FFurkan Gözükara SECourses i never got good results with Prodigy

mikemenders•10/9/23, 1:14 PM

I have now made a perfect Lora with Prodigy, but you have to use Clip Skip 2 when generating and then it will be flexible.

Mmikemenders I have now made a perfect Lora with Prodigy, but you have to use Clip Skip 2 whe...

Furkan Gözükara SECourses•10/9/23, 1:20 PM

interesting never tested clip skip 2. this is sd 1.5?

SStefanoid9343 Thank you so much to @Dr. Furkan Gözükara and @Sandstormleader after several hou...

Furkan Gözükara SECourses•10/9/23, 1:20 PM

awesome

FFurkan Gözükara SECourses interesting never tested clip skip 2. this is sd 1.5?

mikemenders•10/9/23, 1:23 PM

Yes, I trained with RealisticVision 5.1, and I generated the images in the same model.

mikemenders•10/9/23, 1:24 PM

If I generated on clip skip 1, the model was inflexible, but on clip skip 2 everything was fine.
And I thought right: I trained it with clip skip 2 and then clip skip 1 can be generated well.

MetaphysixOP•10/9/23, 1:31 PM

So now the SDXL training works properly with TE, but what's up with 1.5 dreambooth?

rafraf•10/9/23, 1:51 PM

what is the best/fastest gpu deal on Runpod to do inference on Auto1111? A100 or A6000? I find A6000 very slow, slower than 4900...

MMetaphysix So now the SDXL training works properly with TE, but what's up with 1.5 dreamboo...

Furkan Gözükara SECourses•10/9/23, 1:51 PM

yes but using kohya

Furkan Gözükara SECourses•10/9/23, 1:52 PM

i think sd 1.5 dreambooth also should work but need good hyper parameter testing. also kohya missing EMA

Rrafraf what is the best/fastest gpu deal on Runpod to do inference on Auto1111? A100 or...

Furkan Gözükara SECourses•10/9/23, 1:52 PM

4090 could be fastest with price ratio

FFurkan Gözükara SECourses 4090 could be fastest with price ratio

rafraf•10/9/23, 1:56 PM

ok, yea, that's what i thought. a6000 takes twice as long to infer

EEyeSpyBekoAI Hey gang, does anyone have experience on downloading safetensor files from runpo...

Knechtruprecht•10/9/23, 2:09 PM

use "runpodctl" to transfer big Data between your desktop and runpod https://github.com/runpod/runpodctl

GitHub

GitHub - runpod/runpodctl: 🧰 | RunPod CLI for pod management

| RunPod CLI for pod management. Contribute to runpod/runpodctl development by creating an account on GitHub.

GitHub - runpod/runpodctl: 🧰 | RunPod CLI for pod management

thesilvermoth•10/9/23, 2:13 PM

I tested 4090 with Kohya on Windows 11 and it was 1.89 it. On Windows 10 it is 1.06 it. Seems like there is a huge driver issue with W11

FFurkan Gözükara SECourses i think sd 1.5 dreambooth also should work but need good hyper parameter testing...

MetaphysixOP•10/9/23, 2:13 PM

I'm speaking about kohyaSS 1.5 dreambooth training. Can I train SD1.5 with TE?

KKnechtruprecht use "runpodctl" to transfer big Data between your desktop and runpod https://git...

Furkan Gözükara SECourses•10/9/23, 2:14 PM

uploading hugging face is even faster

Agree I made some test for speed/memory. No gradient checkpointing: xformers + Mem Efficient Attenti

2023-10-07

Similar Threads

2023-10-07