Software Engineering Courses (SECourses)•2y ago

Can I do multi-gpu training in Koyha if the graphics cards are different models/different Vram? (exa

Can I do multi-gpu training in Koyha if the graphics cards are different models/different Vram? (example, RTX 3080 and RTX 3090). Sorry to ask before testing, it's that I need a new power supply if I want to use both cards.

FFurkan Gözükara SECourses what is your it / s ?

Alexm•12/29/23, 6:42 PM

it is around 2s/it, a 3090ti has 1.12s/it on the same settings

Alexm•12/29/23, 6:42 PM

the only time when the 4090 is faster, is if i increase the batch size to 4, its around 30% faster than the 3090ti then. But its still a bit weird

Lorelai AI•12/29/23, 6:55 PM

Does anyone know the Syntax for the regional prompter well. I'm getting better with prompts but if I have two Character Loras what's the best way to set up the prompt?

Digital [Starburst]•12/29/23, 9:03 PM

I'm getting [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::resizePhysical::140] Error Code 2: OutofMemory when attempting to generate the TensorRT Default Engine

Digital [Starburst]•12/29/23, 9:04 PM

It also says the Engine is invalid and I need to verify the engine was built correctly

Digital [Starburst]•12/29/23, 9:05 PM

I'm using your automated installers from Patreon and am in Development mode until the install is complete and then switch back to Master.

Digital [Starburst]•12/29/23, 9:09 PM

I checked my CUDA install and it is set to use version 11.8 which I assume is correct.

Digital [Starburst]•12/29/23, 9:11 PM

I have 12.3 installed, but it is not set in Path except that CUDA_PATH_V12_3 is set to that version. I installed 12.3 because it was the only way to check the samples with Visual Studio 2022.

Digital [Starburst]•12/29/23, 9:11 PM

Once I tested it, and it worked I changed the environment variables to 11.8.

Digital [Starburst]•12/29/23, 9:31 PM

Updating torch, torchvision, and xformers tossed another error saying torch could not access the GPU. It said I can set a flag, so it won't run the check, but I don't want to do that.

Digital [Starburst]•12/29/23, 9:31 PM

I am just rerunning your install program to fix it.

Digital [Starburst]•12/29/23, 9:34 PM

Your install program reinstalled cu118, so I'm assuming it must be that version which means I'm running the correct version of CUDA, so I don't think TensorRT is working at the moment.

Digital [Starburst]•12/29/23, 9:43 PM

So it is saying UNSUPPORTED_STATESkipping tactic...insufficient memory. Is there a launch setting such as --medvram I need to use? I haven't tried that yet, but that's next.

Digital [Starburst]•12/29/23, 9:44 PM

It would make sense. The error keeps stating [virtualMemoryBuffer.cpp...etc

Digital [Starburst]•12/29/23, 9:47 PM

Yeah I think that's the issue. The message ends with OUT_OF_MEMORY (No further information). It has to be running out of VRAM.

Digital [Starburst]•12/29/23, 9:54 PM

It is telling me what the problem is, but I'm not seeing it for some reason. Reading the error generated more carefully made it clear, but I'm not sure what to do about it. I set the launcher to --lowvram even though I have 12GB of it, but that still may not be enough. You recommended we choose a machine with at least 24GB of VRAM on RunPod if we are using it.

Digital [Starburst]•12/29/23, 9:57 PM

I was following your tutorial for Kohya, but if I can't even load Tensor then I highly doubt I can run that. The tutorial mentions it can be run on 12GB of VRAM, but I probably won't struggle with it to get it to work.

Digital [Starburst]•12/29/23, 10:09 PM

I couldn't fix it. Whatever is wrong, it is running out of memory before it can finish building the engine. At this point I don't think it really does much good to install it.

DJRich•12/29/23, 11:19 PM

Hi, I am a Patreon subscriber and have been learning how to train SDXL Lora's using the video posted on YouTube. I have trained 4 Loras today. Two of them are fantastic and two are not. The resulting images are high quality, but I noticed that with two of the Loras, the character likeness is only good if the prompt only contains the instance and class prompt. If I add other words to the prompt such as describing clothing, hair, glasses, etc. the likeness drops considerably. I've been using 30 or so images of the subject set to 20 repeats in Kohya and 25 regularization (using the images from the Patreon post) images per Training Image repeat, so in the case of my most recent training I used 700 regularization images since I am repeating them just once. For epochs I have been setting it to a number that equals between 6000-9000 total training steps. I can provide sample images, but I have a feeling I'm missing something easy. Any ideas would be appreciated. I'm happy to post my preset .json for Kohya if it would help. Thanks!

XXylber Can I do multi-gpu training in Koyha if the graphics cards are different models/...

Furkan Gözükara SECourses•12/29/23, 11:40 PM

it loads full model on both GPU. but i don't know how sync works

Furkan Gözükara SECourses•12/29/23, 11:40 PM

so need testing

AAlexm it is around 2s/it, a 3090ti has 1.12s/it on the same settings

Furkan Gözükara SECourses•12/29/23, 11:40 PM

wow that is wrong

DDigital [Starburst]I couldn't fix it. Whatever is wrong, it is running out of memory before it can...

Furkan Gözükara SECourses•12/29/23, 11:41 PM

have you tested my auto installer?

Furkan Gözükara SECourses•12/29/23, 11:41 PM

also what is your gpu?

DDJRich Hi, I am a Patreon subscriber and have been learning how to train SDXL Lora's us...

Furkan Gözükara SECourses•12/29/23, 11:41 PM

hi welcome

Furkan Gözükara SECourses•12/29/23, 11:41 PM

what is your GPU?

FFurkan Gözükara SECourses what is your GPU?

DJRich•12/29/23, 11:42 PM

mobile RTX4090, so 16GB of VRAM

DDJRich mobile RTX4090, so 16GB of VRAM

Furkan Gözükara SECourses•12/29/23, 11:44 PM

i see

Furkan Gözükara SECourses•12/29/23, 11:44 PM

I would also do SDXL DreamBooth training with 16 GB config we have

Furkan Gözükara SECourses•12/29/23, 11:44 PM

and compare results

Furkan Gözükara SECourses•12/29/23, 11:44 PM

it could be better than LoRA

Furkan Gözükara SECourses•12/29/23, 11:44 PM

also training dataset matters a lot in terms of training quality

Furkan Gözükara SECourses•12/29/23, 11:44 PM

that could be difference reason

FFurkan Gözükara SECourses and compare results

DJRich•12/29/23, 11:46 PM

DreamBooth is definitely better, but I like the flexibility of using other models as well as the disk space savings. The image quality results are good from all four Loras, it is just that two of them essentially diminish the subject's likeness as more tokens are added.

DDJRich DreamBooth is definitely better, but I like the flexibility of using other model...

Furkan Gözükara SECourses•12/29/23, 11:47 PM

ok do this

Furkan Gözükara SECourses•12/29/23, 11:47 PM

dreambooth train

Furkan Gözükara SECourses•12/29/23, 11:47 PM

extract lora

Furkan Gözükara SECourses•12/29/23, 11:47 PM

it works great too

FFurkan Gözükara SECourses extract lora

DJRich•12/29/23, 11:48 PM

I thought I had read that the Lora extraction was deprecated in the backend code, but still in the UI. Are there any special settings that should be used for a successful extraction?

DDJRich I thought I had read that the Lora extraction was deprecated in the backend code...

Furkan Gözükara SECourses•12/29/23, 11:49 PM

it works great in kohya gui

Furkan Gözükara SECourses•12/29/23, 11:49 PM

i just tested

Furkan Gözükara SECourses•12/29/23, 11:49 PM

try different ranks

Furkan Gözükara SECourses•12/29/23, 11:49 PM

such as

Furkan Gözükara SECourses•12/29/23, 11:49 PM

128x128

Furkan Gözükara SECourses•12/29/23, 11:49 PM