Software Engineering Courses (SECourses)•2y ago

I'm training on an rtx 3090 with Gradient checkpointing off and CrossAttention sdpa at 1.52s/it

Ddsienra I'm training on an rtx 3090 with Gradient checkpointing off and CrossAttention s...

Furkan Gözükara SECourses•5/21/24, 2:31 AM

nice speed

Furkan Gözükara SECourses•5/21/24, 2:31 AM

spda speed up

Furkan Gözükara SECourses•5/21/24, 2:31 AM

but i dont know how much quality loss

dsienraOP•5/21/24, 2:33 AM

1.51s/it now, still slower than previous versions with the same settings but it is faster than using Gradient checkpointing on this capture is with an older version

FFurkan Gözükara SECourses nice speed

dsienraOP•5/21/24, 2:35 AM

I didn't know that this option can decrease quality, what about xformers, with it also can get quality loss?

dsienraOP•5/21/24, 2:37 AM

I get almost the same speeds with xformers and a little more vram usage 20gb or so

Ddsienra I didn't know that this option can decrease quality, what about xformers, with i...

Furkan Gözükara SECourses•5/21/24, 3:08 AM

yes in past both degraded

Furkan Gözükara SECourses•5/21/24, 3:08 AM

but it is getting better

dsienraOP•5/21/24, 3:17 AM

ok thanks!! I will check if I can see any difference

Furkan Gözükara SECourses•5/21/24, 3:18 AM

i will do a training with onetrainer

Furkan Gözükara SECourses•5/21/24, 3:18 AM

lets see difference between kohya

dsienraOP•5/21/24, 3:23 AM

that will be an interesting test!! a good test can be also to do a run test with xformers and spda, its a little speed up but can be very significant speed up for very large datasets and very long trainings.

Furkan Gözükara SECourses•5/21/24, 3:25 AM

doing a fresh install

Furkan Gözükara SECourses•5/21/24, 3:26 AM

of onetrainer

Furkan Gözükara SECourses•5/21/24, 3:26 AM

ye for massive dataset

Furkan Gözükara SECourses•5/21/24, 3:26 AM

you have to have xformers and batch size bigger

NNicB Yes. I'll work on getting that fixed tomorrow with the team

RMK•5/21/24, 3:27 AM

Hi just checking to see if this works now ? And if so, how do I do it?

dsienraOP•5/21/24, 3:31 AM

LR Scheduler sdpa never worked on kohya_ss an sdxl models, seems to be an interesting concept I read that it automatically adjusts the learning rate or something like that.

dsienraOP•5/21/24, 3:32 AM

I don't know if it was designed to work in conjunction with CrossAttention sdpa

Ddsienra LR Scheduler sdpa never worked on kohya_ss an sdxl models, seems to be an intere...

Furkan Gözükara SECourses•5/21/24, 3:36 AM

i use static LR

FFurkan Gözükara SECourses i use static LR

dsienraOP•5/21/24, 3:41 AM

Yes I know, I think is the best choice for a little dataset and for training just one subject, but what do you think for other cases like bigger and probably unbalanced datasets with multiple subjects? a smart lr could be nice.

Narva•5/21/24, 3:43 AM

I have trained a lora model in kohya and I have obtained better likeness and consistency than with a full dreambooth model

Narva•5/21/24, 3:43 AM

Lora is a underrated model

Ddsienra Yes I know, I think is the best choice for a little dataset and for training jus...

Furkan Gözükara SECourses•5/21/24, 3:44 AM

smart LR not working for stable diffusion

Furkan Gözükara SECourses•5/21/24, 3:44 AM

they work for LLMs

NNarva I have trained a lora model in kohya and I have obtained better likeness and con...

Furkan Gözükara SECourses•5/21/24, 3:44 AM

probably your config was not good for dreambooth. lora is scientifically proven to be inferior

FFurkan Gözükara SECourses probably your config was not good for dreambooth. lora is scientifically proven ...

Narva•5/21/24, 3:44 AM

I used your presets

Furkan Gözükara SECourses•5/21/24, 3:45 AM

https://www.linkedin.com/posts/furkangozukara_why-i-dont-research-lora-training-because-activity-7164700874097856512--Ecj/?utm_source=share&utm_medium=member_desktop

Furkan Gözükara on LinkedIn: Why I don't research LoRA training? Be...

Why I don't research LoRA training? Because if you need LoRA, do a full DreamBooth / Fine Tuning and extract LoRA. You will get better quality. This is for…

NNarva I used your presets 😅

Furkan Gözükara SECourses•5/21/24, 3:47 AM

onetrianer preset published in this video : https://youtu.be/0t5l6CP9eBg

YouTubeSECourses

Full Stable Diffusion SD & XL Fine Tuning Tutorial With OneTrainer ...

In this tutorial, I am going to show you how to install OneTrainer from scratch on your computer and do a Stable Diffusion SDXL (Full Fine-Tuning 10.3 GB VRAM) and SD 1.5 (Full Fine-Tuning 7GB VRAM) based models training on your computer and also do the same training on a very cheap cloud machine from MassedCompute if you don't have such compute...

FFurkan Gözükara SECourses onetrianer preset published in this video : https://youtu.be/0t5l6CP9eBg

Narva•5/21/24, 3:47 AM

I did too, I watched and trained with onetrainer

Narva•5/21/24, 3:47 AM

super weird

FFurkan Gözükara SECourses smart LR not working for stable diffusion 😄

dsienraOP•5/21/24, 3:48 AM

ohh ok I didn't know, I'm not a programmer

Narva•5/21/24, 3:48 AM

Use portrait photos taken with a sony camera

Narva•5/21/24, 3:49 AM

I used my photography lights to take the photos and stuff

Narva•5/21/24, 3:49 AM

But it didn't give me a good enough likenss

Narva•5/21/24, 3:50 AM

I'm doing a comparison, I'll send them later too see

NNarva Use portrait photos taken with a sony camera

dsienraOP•5/21/24, 3:51 AM

With the right parameters and number of steps full checkpoint is better, but depending your dataset number of images, etc may be it needs more steps or something

Ddsienra With the right parameters and number of steps full checkpoint is better, but de...

Narva•5/21/24, 3:52 AM

I know, I used just kind 16 portrait photos, also realistic vision model

dsienraOP•5/21/24, 3:55 AM

I'm not an expert but I'm talking from my own experience, I had very good results with lora too I'm not gonna lie but in my opinion Dr. Furkan Gözükara is right in this one

Narva•5/21/24, 3:55 AM

Yeah is right

Narva•5/21/24, 4:00 AM

It seems curious to me that I spent a lot of time training and spending money training full dreambooth and I got good results but with just one time that I did lora training, it gave me very good results too

dsienraOP•5/21/24, 4:07 AM

Lora is faster easier and it works fine for one purpose, but if you want to train a bigger dataset or many people at the same time full training is better, I had mixed results with both methods but I kept trying with different settings trainers and trainer versions, some with good results and some disappointing ones

Poppy•5/21/24, 8:11 AM

Hey, does anyone know how to tell when a file has successfully uploaded into JupyterLab? I've deployed a pod with comfyui and have uploaded a lora into the lora folder, however in http, the lora step has an error. Is there any way to know when the lora has fully loaded in to jupyterlab?

PPoppy Hey, does anyone know how to tell when a file has successfully uploaded into Jup...

TheToday•5/21/24, 8:13 AM

If it is in runpod below you get like a box with the progress

TTheToday If it is in runpod below you get like a box with the progress

Poppy•5/21/24, 8:24 AM

Thank you!

FFurkan Gözükara SECourses yes in past both degraded

dsienraOP•5/21/24, 8:32 AM

I was training with your new parameters, 5 people different genders and ages, no regularization images, on 30 repetitions the resemblase of the people started to get good with a simple prompt and photorealistic, but I tried some other prompts with other styles some cyberpunk and some of your prompts and they started to look baked worse and worse during the training, I was using the base model, now I stopped the training and disabled the spda optimizer and I started again, lets see..... may be is that... any idea?
I used the base model to have better compatibility with loras but if the problem continues I will try with RealVis XL 4, I'm attaching some images look ok at the beginning of the training and the baked ones with 30 repetitions

Poppy•5/21/24, 9:18 AM

does anyone know if there's a faster way to upload loras from the pc to runpod? At the moment it's taking around 40 minutes per lora

PPoppy does anyone know if there's a faster way to upload loras from the pc to runpod? ...

TheToday•5/21/24, 9:40 AM

Upload to huggingface and download in runpod

FFurkan Gözükara SECourses Kohya config updated https://www.patreon.com/posts/very-best-for-of-89213064

gohar•5/21/24, 9:54 AM

Can we expect detailed youtube tutorial of this like onetrainer soon?

hmm•5/21/24, 11:02 AM