Software Engineering Courses (SECourses)•2y ago

oohhhh 🙂 What changes?

oohhhh

What changes?

DigitalArt_Lover•12/29/23, 8:24 AM

Oh, i can just edit the bat file and add autoupdate, it should update automatically, everytime it runs?

LLorelai AI oohhhh 🙂 What changes?

Benjamin•12/29/23, 8:24 AM

https://github.com/bmaltais/kohya_ss/releases/tag/v22.4.0

GitHub

Release v22.4.0 · bmaltais/kohya_ss

2023/12/28 (v22.4.0)

Fixed to work tools/convert_diffusers20_original_sd.py. Thanks to Disty0! PR #1016
The issues in multi-GPU training are fixed. Thanks to Isotr0py! PR #989 and #1000

--ddp_gr...

Lorelai AIOP•12/29/23, 8:24 AM

Ohhh seems useless unless you're running more than one GPU

Benjamin•12/29/23, 8:25 AM

PagedAdamW added as optimizer and for the rest mostly fixes, but also some new options for sampling

Lorelai AIOP•12/29/23, 8:26 AM

Ok now im interested, whats pagedadamw

Lorelai AIOP•12/29/23, 8:27 AM

ok yeah im about to do a training it looks interesting

Lorelai AIOP•12/29/23, 8:35 AM

WWhat's the command to update Kohya again I forgot?

Lorelai AIOP•12/29/23, 8:37 AM

nevermind just a git pull

Lorelai AIOP•12/29/23, 8:38 AM

Let's hope nothing is broken now

Lorelai AIOP•12/29/23, 8:54 AM

It broke blip ughhhh

Lorelai AIOP•12/29/23, 9:13 AM

The only thing that broke, it seems, was Blip captioning, but as I recall, I was using some type of rollback version of something. I think it was transformers. Honestly most of these captions tools are lousy and if you are using a smaller data-set manual captioning is best

FFurkan Gözükara SECourses Click to see attachment

magical•12/29/23, 1:11 PM

does this mean we can do BF16 training now?

Ren•12/29/23, 1:46 PM

Can I train a lora model for sdxl-turbo?

KKallamamran @Dr. Furkan Gözükara Why 1536x1536 pixel images in the datasets? Training above ...

Furkan Gözükara SECourses•12/29/23, 2:26 PM

yes not but i still put if someone tries or uses

AAlexm just wanted to ask if some people on here use 4090s for dreambooth training in k...

Furkan Gözükara SECourses•12/29/23, 2:27 PM

what is your it / s ?

Bbanfi6885 Hello Furkan, I have watched this video a couple of times and followed the instr...

Furkan Gözükara SECourses•12/29/23, 2:27 PM

hello. are you installing on activated venv?

Furkan Gözükara SECourses•12/29/23, 2:27 PM

which app_

AAlexm is it possible to train dreambooth on kohya with 24gig of vram? no matter what i...

Furkan Gözükara SECourses•12/29/23, 2:27 PM

yes i am doing that all the time. uses around 17 gb

Aallen @Dr. Furkan Gözükara Hey there - it seems there is an issue with download the r...

Furkan Gözükara SECourses•12/29/23, 2:28 PM

true please download attachments. they got updated

Aallen @Dr. Furkan Gözükara aha, it's in the zip file at the end. maybe remove the 3 ap...

Furkan Gözükara SECourses•12/29/23, 2:29 PM

true now in those zips

DDigitalArt_Lover can anyone, help me fix this?? maybe this is the reason, my gpu isnt using full ...

Furkan Gözükara SECourses•12/29/23, 2:29 PM

no it is not an issue. also you can upgrade kohya to latest

Furkan Gözükara SECourses•12/29/23, 2:29 PM

it is not currently at you

Mmagical does this mean we can do BF16 training now?

Furkan Gözükara SECourses•12/29/23, 2:30 PM

sadly not . it is gpu type limitation

RRen Can I train a lora model for sdxl-turbo?

Furkan Gözükara SECourses•12/29/23, 2:30 PM

i dont know yet sadly

FFurkan Gözükara SECourses sadly not . it is gpu type limitation

magical•12/29/23, 2:31 PM

thank you to use both gpus should we add "-ddp_gradient_as_bucket_view" to additional parameters?

magical•12/29/23, 2:32 PM

The issues in multi-GPU training are fixed. Thanks to Isotr0py! PR #989 and #1000

--ddp_gradient_as_bucket_view and --ddp_bucket_viewoptions are added to sdxl_train.py. Please specify these options for multi-GPU training.

magical•12/29/23, 2:33 PM

how do we specify these options in kohya?

Mmagical thank you to use both gpus should we add "-ddp_gradient_as_bucket_view" to addit...

Furkan Gözükara SECourses•12/29/23, 2:35 PM

yes i think

Furkan Gözükara SECourses•12/29/23, 2:35 PM

extra parameters

Xylber•12/29/23, 6:18 PM

Can I do multi-gpu training in Koyha if the graphics cards are different models/different Vram? (example, RTX 3080 and RTX 3090). Sorry to ask before testing, it's that I need a new power supply if I want to use both cards.

FFurkan Gözükara SECourses what is your it / s ?

Alexm•12/29/23, 6:42 PM

it is around 2s/it, a 3090ti has 1.12s/it on the same settings

Alexm•12/29/23, 6:42 PM

the only time when the 4090 is faster, is if i increase the batch size to 4, its around 30% faster than the 3090ti then. But its still a bit weird

Lorelai AIOP•12/29/23, 6:55 PM

Does anyone know the Syntax for the regional prompter well. I'm getting better with prompts but if I have two Character Loras what's the best way to set up the prompt?

Digital [Starburst]•12/29/23, 9:03 PM

I'm getting [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::resizePhysical::140] Error Code 2: OutofMemory when attempting to generate the TensorRT Default Engine

Digital [Starburst]•12/29/23, 9:04 PM

It also says the Engine is invalid and I need to verify the engine was built correctly

Digital [Starburst]•12/29/23, 9:05 PM

I'm using your automated installers from Patreon and am in Development mode until the install is complete and then switch back to Master.

Digital [Starburst]•12/29/23, 9:09 PM

I checked my CUDA install and it is set to use version 11.8 which I assume is correct.

Digital [Starburst]•12/29/23, 9:11 PM

I have 12.3 installed, but it is not set in Path except that CUDA_PATH_V12_3 is set to that version. I installed 12.3 because it was the only way to check the samples with Visual Studio 2022.

Digital [Starburst]•12/29/23, 9:11 PM

Once I tested it, and it worked I changed the environment variables to 11.8.

Digital [Starburst]•12/29/23, 9:31 PM

Updating torch, torchvision, and xformers tossed another error saying torch could not access the GPU. It said I can set a flag, so it won't run the check, but I don't want to do that.

Digital [Starburst]•12/29/23, 9:31 PM

I am just rerunning your install program to fix it.

Digital [Starburst]•12/29/23, 9:34 PM

Your install program reinstalled cu118, so I'm assuming it must be that version which means I'm running the correct version of CUDA, so I don't think TensorRT is working at the moment.

Digital [Starburst]•12/29/23, 9:43 PM

So it is saying UNSUPPORTED_STATESkipping tactic...insufficient memory. Is there a launch setting such as --medvram I need to use? I haven't tried that yet, but that's next.

Digital [Starburst]•12/29/23, 9:44 PM

It would make sense. The error keeps stating [virtualMemoryBuffer.cpp...etc

Digital [Starburst]•12/29/23, 9:47 PM

Yeah I think that's the issue. The message ends with OUT_OF_MEMORY (No further information). It has to be running out of VRAM.

Digital [Starburst]•12/29/23, 9:54 PM

It is telling me what the problem is, but I'm not seeing it for some reason. Reading the error generated more carefully made it clear, but I'm not sure what to do about it. I set the launcher to --lowvram even though I have 12GB of it, but that still may not be enough. You recommended we choose a machine with at least 24GB of VRAM on RunPod if we are using it.

Digital [Starburst]•12/29/23, 9:57 PM

I was following your tutorial for Kohya, but if I can't even load Tensor then I highly doubt I can run that. The tutorial mentions it can be run on 12GB of VRAM, but I probably won't struggle with it to get it to work.

Digital [Starburst]•12/29/23, 10:09 PM

I couldn't fix it. Whatever is wrong, it is running out of memory before it can finish building the engine. At this point I don't think it really does much good to install it.

DJRich•12/29/23, 11:19 PM

Hi, I am a Patreon subscriber and have been learning how to train SDXL Lora's using the video posted on YouTube. I have trained 4 Loras today. Two of them are fantastic and two are not. The resulting images are high quality, but I noticed that with two of the Loras, the character likeness is only good if the prompt only contains the instance and class prompt. If I add other words to the prompt such as describing clothing, hair, glasses, etc. the likeness drops considerably. I've been using 30 or so images of the subject set to 20 repeats in Kohya and 25 regularization (using the images from the Patreon post) images per Training Image repeat, so in the case of my most recent training I used 700 regularization images since I am repeating them just once. For epochs I have been setting it to a number that equals between 6000-9000 total training steps. I can provide sample images, but I have a feeling I'm missing something easy. Any ideas would be appreciated. I'm happy to post my preset .json for Kohya if it would help. Thanks!

XXylber Can I do multi-gpu training in Koyha if the graphics cards are different models/...

Furkan Gözükara SECourses•12/29/23, 11:40 PM

it loads full model on both GPU. but i don't know how sync works

oohhhh 🙂 What changes?

Similar Threads

Similar Threads

Similar Threads