Software Engineering Courses (SECourses)•2y ago

just wanted to ask if some people on here use 4090s for dreambooth training in kohya and their exper

just wanted to ask if some people on here use 4090s for dreambooth training in kohya and their experience? I noticed that on runpod, a 4090 is a lot slower than a 3090ti and im trying to figure out if this is a issue on runpod or with a 4090 overall. Ive seen some issues posted on the kohya github about 4090s being slow, but im not sure if this is still the case

FFurkan Gözükara SECourses must watch video : https://youtu.be/-NjNy7afOQ0

banfi6885•12/28/23, 11:22 PM

Hello Furkan, I have watched this video a couple of times and followed the instructions.

I have installed torch using the following command:
pip3 install torch==2.1.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

and transformers using the following command:
pip install xformers==0.0.23

I also edit webui-user.bat to use xformers:
set COMMANDLINE_ARGS=--xformers

With pip install xformers==0.0.23, I get the following:

Installing collected packages: torch, xformers
Attempting uninstall: torch
Found existing installation: torch 2.1.0+cu118
Uninstalling torch-2.1.0+cu118:
Successfully uninstalled torch-2.1.0+cu118
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.16.0+cu118 requires torch==2.1.0+cu118, but you have torch 2.1.1 which is incompatible.
torchaudio 2.1.0+cu118 requires torch==2.1.0+cu118, but you have torch 2.1.1 which is incompatible.
Successfully installed torch-2.1.1 xformers-0.0.23

And when I execute webiu-user.bat, I get the following:
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

Is there something I am overlooking?

Thank you

AlexmOP•12/28/23, 11:38 PM

is it possible to train dreambooth on kohya with 24gig of vram? no matter what i try, it keeps running oom with 8bitadam

allen•12/29/23, 3:20 AM

@Dr. Furkan Gözükara Hey there - it seems there is an issue with download the runpod installer for your caption scripts

allen•12/29/23, 3:20 AM

allen•12/29/23, 3:20 AM

https://www.patreon.com/posts/sota-very-best-90744385

AAlexm is it possible to train dreambooth on kohya with 24gig of vram? no matter what i...

allen•12/29/23, 3:22 AM

are you using the runpod kohya template by any chance? i had never noticed until watching the good dr's video that if you don't kill auto1111, 25-30% of the vram is getting sucked up before you start

allen•12/29/23, 3:23 AM

i typically just reduce the batch size until it works, or throw on xformers, but i'm going to retry a few of them without cheating since apparently its bad juju

allen•12/29/23, 3:29 AM

@Dr. Furkan Gözükara aha, it's in the zip file at the end. maybe remove the 3 apparent links in that article when you have a sec

DigitalArt_Lover•12/29/23, 7:58 AM

can anyone, help me fix this??
maybe this is the reason, my gpu isnt using full potential, while running the training!!

DDigitalArt_Lover can anyone, help me fix this?? maybe this is the reason, my gpu isnt using full ...

Lorelai AI•12/29/23, 8:09 AM

Looks at potential to me, what are you looking at? You should update Kohya to 22.3 though

LLorelai AI Looks at potential to me, what are you looking at? You should update Kohya to 22...

DigitalArt_Lover•12/29/23, 8:22 AM

Oh, i see, yeah, let me do it as well. potential, means, when i run training, it doesnt take load on gpu it should take....usually, when the temperature gets higher, when the gpu is totally using it's power, but when i run training, with kohya, it only gets half of the temperature, compared to it usually takes.

LLorelai AI Looks at potential to me, what are you looking at? You should update Kohya to 22...

Benjamin•12/29/23, 8:23 AM

since today there's 22.4

BBenjamin since today there's 22.4

Lorelai AI•12/29/23, 8:23 AM

oohhhh

What changes?

DigitalArt_Lover•12/29/23, 8:24 AM

Oh, i can just edit the bat file and add autoupdate, it should update automatically, everytime it runs?

LLorelai AI oohhhh 🙂 What changes?

Benjamin•12/29/23, 8:24 AM

https://github.com/bmaltais/kohya_ss/releases/tag/v22.4.0

GitHub

Release v22.4.0 · bmaltais/kohya_ss

2023/12/28 (v22.4.0)

Fixed to work tools/convert_diffusers20_original_sd.py. Thanks to Disty0! PR #1016
The issues in multi-GPU training are fixed. Thanks to Isotr0py! PR #989 and #1000

--ddp_gr...

Lorelai AI•12/29/23, 8:24 AM

Ohhh seems useless unless you're running more than one GPU

Benjamin•12/29/23, 8:25 AM

PagedAdamW added as optimizer and for the rest mostly fixes, but also some new options for sampling

Lorelai AI•12/29/23, 8:26 AM

Ok now im interested, whats pagedadamw

Lorelai AI•12/29/23, 8:27 AM

ok yeah im about to do a training it looks interesting

Lorelai AI•12/29/23, 8:35 AM

WWhat's the command to update Kohya again I forgot?

Lorelai AI•12/29/23, 8:37 AM

nevermind just a git pull

Lorelai AI•12/29/23, 8:38 AM

Let's hope nothing is broken now

Lorelai AI•12/29/23, 8:54 AM

It broke blip ughhhh

Lorelai AI•12/29/23, 9:13 AM

The only thing that broke, it seems, was Blip captioning, but as I recall, I was using some type of rollback version of something. I think it was transformers. Honestly most of these captions tools are lousy and if you are using a smaller data-set manual captioning is best

FFurkan Gözükara SECourses Click to see attachment

magical•12/29/23, 1:11 PM

does this mean we can do BF16 training now?

Ren•12/29/23, 1:46 PM

Can I train a lora model for sdxl-turbo?

KKallamamran @Dr. Furkan Gözükara Why 1536x1536 pixel images in the datasets? Training above ...

Furkan Gözükara SECourses•12/29/23, 2:26 PM

yes not but i still put if someone tries or uses

AAlexm just wanted to ask if some people on here use 4090s for dreambooth training in k...

Furkan Gözükara SECourses•12/29/23, 2:27 PM

what is your it / s ?

Bbanfi6885 Hello Furkan, I have watched this video a couple of times and followed the instr...

Furkan Gözükara SECourses•12/29/23, 2:27 PM

hello. are you installing on activated venv?

Furkan Gözükara SECourses•12/29/23, 2:27 PM

which app_

AAlexm is it possible to train dreambooth on kohya with 24gig of vram? no matter what i...

Furkan Gözükara SECourses•12/29/23, 2:27 PM

yes i am doing that all the time. uses around 17 gb

Aallen @Dr. Furkan Gözükara Hey there - it seems there is an issue with download the r...

Furkan Gözükara SECourses•12/29/23, 2:28 PM

true please download attachments. they got updated

Aallen @Dr. Furkan Gözükara aha, it's in the zip file at the end. maybe remove the 3 ap...

Furkan Gözükara SECourses•12/29/23, 2:29 PM

true now in those zips

DDigitalArt_Lover can anyone, help me fix this?? maybe this is the reason, my gpu isnt using full ...

Furkan Gözükara SECourses•12/29/23, 2:29 PM

no it is not an issue. also you can upgrade kohya to latest

Furkan Gözükara SECourses•12/29/23, 2:29 PM

it is not currently at you

Mmagical does this mean we can do BF16 training now?

Furkan Gözükara SECourses•12/29/23, 2:30 PM

sadly not . it is gpu type limitation

RRen Can I train a lora model for sdxl-turbo?

Furkan Gözükara SECourses•12/29/23, 2:30 PM

i dont know yet sadly

FFurkan Gözükara SECourses sadly not . it is gpu type limitation

magical•12/29/23, 2:31 PM

thank you to use both gpus should we add "-ddp_gradient_as_bucket_view" to additional parameters?

magical•12/29/23, 2:32 PM

The issues in multi-GPU training are fixed. Thanks to Isotr0py! PR #989 and #1000

--ddp_gradient_as_bucket_view and --ddp_bucket_viewoptions are added to sdxl_train.py. Please specify these options for multi-GPU training.

magical•12/29/23, 2:33 PM

how do we specify these options in kohya?

Mmagical thank you to use both gpus should we add "-ddp_gradient_as_bucket_view" to addit...

Furkan Gözükara SECourses•12/29/23, 2:35 PM

yes i think

Furkan Gözükara SECourses•12/29/23, 2:35 PM

extra parameters

Xylber•12/29/23, 6:18 PM

Can I do multi-gpu training in Koyha if the graphics cards are different models/different Vram? (example, RTX 3080 and RTX 3090). Sorry to ask before testing, it's that I need a new power supply if I want to use both cards.

FFurkan Gözükara SECourses what is your it / s ?

AlexmOP•12/29/23, 6:42 PM

it is around 2s/it, a 3090ti has 1.12s/it on the same settings

AlexmOP•12/29/23, 6:42 PM

the only time when the 4090 is faster, is if i increase the batch size to 4, its around 30% faster than the 3090ti then. But its still a bit weird

Lorelai AI•12/29/23, 6:55 PM

Does anyone know the Syntax for the regional prompter well. I'm getting better with prompts but if I have two Character Loras what's the best way to set up the prompt?

Digital [Starburst]•12/29/23, 9:03 PM

I'm getting [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::resizePhysical::140] Error Code 2: OutofMemory when attempting to generate the TensorRT Default Engine

Digital [Starburst]•12/29/23, 9:04 PM

It also says the Engine is invalid and I need to verify the engine was built correctly

Digital [Starburst]•12/29/23, 9:05 PM

I'm using your automated installers from Patreon and am in Development mode until the install is complete and then switch back to Master.

Digital [Starburst]•12/29/23, 9:09 PM

I checked my CUDA install and it is set to use version 11.8 which I assume is correct.

just wanted to ask if some people on here use 4090s for dreambooth training in kohya and their exper

Similar Threads

Similar Threads

Similar Threads