thank you to use both gpus should we add "-ddp_gradient_as_bucket_view" to additional parameters?
thank you to use both gpus should we add "-ddp_gradient_as_bucket_view" to additional parameters?
MThe issues in multi-GPU training are fixed. Thanks to Isotr0py! PR #989 and #1000
--ddp_gradient_as_bucket_view and --ddp_bucket_viewoptions are added to sdxl_train.py. Please specify these options for multi-GPU training.
--ddp_gradient_as_bucket_view and --ddp_bucket_viewoptions are added to sdxl_train.py. Please specify these options for multi-GPU training.
Mhow do we specify these options in kohya?
Fyes i think
Fextra parameters
XCan I do multi-gpu training in Koyha if the graphics cards are different models/different Vram? (example, RTX 3080 and RTX 3090). Sorry to ask before testing, it's that I need a new power supply if I want to use both cards.
Ait is around 2s/it, a 3090ti has 1.12s/it on the same settings
Athe only time when the 4090 is faster, is if i increase the batch size to 4, its around 30% faster than the 3090ti then. But its still a bit weird
LDoes anyone know the Syntax for the regional prompter well. I'm getting better with prompts but if I have two Character Loras what's the best way to set up the prompt?
![Digital [Starburst]](https://cdn.discordapp.com/avatars/220961909447458817/8ecbbb3132fc6e44312f3d6d2ae3df69.webp?size=40)
DI'm getting [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::resizePhysical::140] Error Code 2: OutofMemory when attempting to generate the TensorRT Default Engine
DIt also says the Engine is invalid and I need to verify the engine was built correctly
DI'm using your automated installers from Patreon and am in Development mode until the install is complete and then switch back to Master.
DI checked my CUDA install and it is set to use version 11.8 which I assume is correct.
DI have 12.3 installed, but it is not set in Path except that CUDA_PATH_V12_3 is set to that version. I installed 12.3 because it was the only way to check the samples with Visual Studio 2022.
DOnce I tested it, and it worked I changed the environment variables to 11.8.
DUpdating torch, torchvision, and xformers tossed another error saying torch could not access the GPU. It said I can set a flag, so it won't run the check, but I don't want to do that.
DI am just rerunning your install program to fix it.
DYour install program reinstalled cu118, so I'm assuming it must be that version which means I'm running the correct version of CUDA, so I don't think TensorRT is working at the moment.
DSo it is saying UNSUPPORTED_STATESkipping tactic...insufficient memory. Is there a launch setting such as --medvram I need to use? I haven't tried that yet, but that's next.
DIt would make sense. The error keeps stating [virtualMemoryBuffer.cpp...etc
DYeah I think that's the issue. The message ends with OUT_OF_MEMORY (No further information). It has to be running out of VRAM.
DIt is telling me what the problem is, but I'm not seeing it for some reason. Reading the error generated more carefully made it clear, but I'm not sure what to do about it. I set the launcher to --lowvram even though I have 12GB of it, but that still may not be enough. You recommended we choose a machine with at least 24GB of VRAM on RunPod if we are using it.
DI was following your tutorial for Kohya, but if I can't even load Tensor then I highly doubt I can run that. The tutorial mentions it can be run on 12GB of VRAM, but I probably won't struggle with it to get it to work.
DI couldn't fix it. Whatever is wrong, it is running out of memory before it can finish building the engine. At this point I don't think it really does much good to install it.
DHi, I am a Patreon subscriber and have been learning how to train SDXL Lora's using the video posted on YouTube. I have trained 4 Loras today. Two of them are fantastic and two are not. The resulting images are high quality, but I noticed that with two of the Loras, the character likeness is only good if the prompt only contains the instance and class prompt. If I add other words to the prompt such as describing clothing, hair, glasses, etc. the likeness drops considerably. I've been using 30 or so images of the subject set to 20 repeats in Kohya and 25 regularization (using the images from the Patreon post) images per Training Image repeat, so in the case of my most recent training I used 700 regularization images since I am repeating them just once. For epochs I have been setting it to a number that equals between 6000-9000 total training steps. I can provide sample images, but I have a feeling I'm missing something easy. Any ideas would be appreciated. I'm happy to post my preset .json for Kohya if it would help. Thanks!
Fit loads full model on both GPU. but i don't know how sync works
Fso need testing
Fwow that is wrong
![Digital [Starburst]](https://cdn.discordapp.com/avatars/220961909447458817/8ecbbb3132fc6e44312f3d6d2ae3df69.webp?size=16)
Fhave you tested my auto installer?
Falso what is your gpu?
Fhi welcome
Fwhat is your GPU?
Dmobile RTX4090, so 16GB of VRAM
Fi see
FI would also do SDXL DreamBooth training with 16 GB config we have
Fand compare results
Fit could be better than LoRA
Falso training dataset matters a lot in terms of training quality
Fthat could be difference reason
DDreamBooth is definitely better, but I like the flexibility of using other models as well as the disk space savings. The image quality results are good from all four Loras, it is just that two of them essentially diminish the subject's likeness as more tokens are added.
Fok do this
Fdreambooth train
Fextract lora
Fit works great too
DI thought I had read that the Lora extraction was deprecated in the backend code, but still in the UI. Are there any special settings that should be used for a successful extraction?
Fit works great in kohya gui
Fi just tested
Ftry different ranks
Fsuch as
F128x128
F192x192
