the kohya GUI doesnt show multi GPU. so when you use 6 GPUs your actual batch size is being 6. it du
the kohya GUI doesnt show multi GPU. so when you use 6 GPUs your actual batch size is being 6. it duplicates training on all
Fso you need to setup your number of epochs or repeating accordingly
VI stopped using the kohya GUI because I couldn't see how to enable multi-gpu, so I've just been using a command line argument. This is the argument:
I've tried this with a batch size of 1 and 2. 2 is actually slower than 1, and anything higher than 2 gives me an out of memory error.
I've gone ahead and tried doing a repeat of 40 on the original images, and now 7 (close enough to 40 / 6).
Changing that didn't change the total number of optimization steps, which was always 9600, or the time to complete training. I think it just wanted to do more epochs.
After changing the repeats to 7 and removing the flag for
I tried using the
I've tried this with a batch size of 1 and 2. 2 is actually slower than 1, and anything higher than 2 gives me an out of memory error.
I've gone ahead and tried doing a repeat of 40 on the original images, and now 7 (close enough to 40 / 6).
Changing that didn't change the total number of optimization steps, which was always 9600, or the time to complete training. I think it just wanted to do more epochs.
After changing the repeats to 7 and removing the flag for
--max_train_steps="9600"I tried using the
--ddp_gradient_as_bucket_viewAfter all this, using just a single 4090 with your default settings gives me a training time of 3 hours @ 8 epochs and 9600 steps versus 4+h on 6 4090s.
Sorry for being a mess! I just don't understand what I'm doing wrong to mess up multi-gpu training so badly. Thank you for all that you're doing to help us though!
Sorry for being a mess! I just don't understand what I'm doing wrong to mess up multi-gpu training so badly. Thank you for all that you're doing to help us though!
Fkohya gui only prepares this command
Faccelerate is the one that does distributed training
Fi can give you private consultation if you need
MAnyway, I'm following the tutorial. It's amazing!
AHow come img2img doesn't work?
Ait just barely changes the color and makes my character ruffled up
Atxt2img works fine for a realistic mouse
S@Dr. Furkan Gözükara sorry for the delay. Heres screenshots of my adetailer settings. I have also used the photo of ohwx man prompt. Also could DM photos if you want to see examples of with/without adetailer
Sin response to this
VThank you for the offer! I was able to discover to 4090 GPUs don't have NVLink, whoops, haha
MGuys just a question, once I've configured everything and run kaggle, is there a way to save the settings after running a session?
Sif you're referring to training, download the .json file and next time you need to use it, upload it. if you're referring to generating images in kaggle, you can upload a png (such as the last image you generated) with the desired settings to "png info" and then send those settings to txt2img
MGood morning! I just wanted to ask for information. To generate images with your own face, use the checkpoint (https://www.youtube.com/watch?v=16-b1AjvyBE&t=1883s), while in this other video, use the lora (https://youtube .com/watch?v=JF2P7BIUpIU&t=1260s)
YouTubeSECourses
Master Stable Diffusion XL Training on Kaggle for Free! Welcome to this comprehensive tutorial where I'll be guiding you through the exciting world of setting up and training Stable Diffusion XL (SDXL) with Kohya on a free Kaggle account. This video is your one-stop resource for learning everything from initiating a Kaggle session with dual ...
Master Stable Diffusion XL Training on Kaggle for Free! Welcome to this comprehensive tutorial where I'll be guiding you through the exciting world of setting up and training Stable Diffusion XL (SDXL) with Kohya on a free Kaggle account. This video is your one-stop resource for learning everything from initiating a Kaggle session with dual ...
Bthese are different training methods
BIve tested a few trainings with your SDXL dreambooth, then lora extraction method, no captions and using reg images but when I generate images using the LORA, it doesnt resemble the subjects face
1Hi there, i get an error in dreambooth when saving the models. It shows a massive error list, that starts with :
Exception training model: ' Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'encoder.conv_in.weight', 'decoder.up_blocks.1.upsamplers.0.conv.weight', 'decoder.up_blocks.3.resnets.0.conv_shortcut.weight', 'decoder.up_blocks.3.resnets.0.norm1.weight', 'encoder.mid_block.resnets.1.norm1.weight',
and ends with:
'decoder.up_blocks.0.resnets.0.conv1.weight', 'decoder.conv_out.weight'}]. A potential way to correctly save your model is to use
I used the fixed github commits that are shown in this video, i used the 2 bat files that are linked on the patreon page.
https://www.youtube.com/watch?v=g0wXIcRhkJk
Exception training model: ' Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'encoder.conv_in.weight', 'decoder.up_blocks.1.upsamplers.0.conv.weight', 'decoder.up_blocks.3.resnets.0.conv_shortcut.weight', 'decoder.up_blocks.3.resnets.0.norm1.weight', 'encoder.mid_block.resnets.1.norm1.weight',
and ends with:
'decoder.up_blocks.0.resnets.0.conv1.weight', 'decoder.conv_out.weight'}]. A potential way to correctly save your model is to use
save_modelI used the fixed github commits that are shown in this video, i used the 2 bat files that are linked on the patreon page.
https://www.youtube.com/watch?v=g0wXIcRhkJk
message.txt11.13KB
YouTubeSECourses
Dreambooth is the best training method for Stable Diffusion. In this tutorial, I show how to install the Dreambooth extension of Automatic1111 Web UI from scratch. Additionally, I demonstrate my months of work on the realism workflow, which enables you to produce studio-quality images of yourself through #Dreambooth training. Furthermore, I shar...
1Just did the patreon btw 
1Oh... i think i found the source of the error. I had the preview images every N epochs activated
TThe same happened for me. Turn the samples creation off down to 0. Also I turned off xtensors (to default) and turned off cache latents. The dreambooth runs pretty fast on my 4070 laptop. Around 1.35 it/s and 8,7gb vram with 512x512 images. Is it ok? Does default scheduler affect quality. With xtensors on it was very slow with much higher vram usage.
AI was thinking about training a model on people wearing my t-shirt design and than use inpainting to put the t-shirts on other people. would that work?
D@Dr. Furkan Gözükara You should make a quick video on this:
Nvidia has released a fix for dealing with Stable Diffusion slowdowns occurring due to Shared Memory feature they introduced some while ago...
https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/11062
https://nvidia.custhelp.com/app/answers/detail/a_id/5490
This was a while back, but I found out today. I believe a lot of people are still on older drivers due to this issue...
Nvidia has released a fix for dealing with Stable Diffusion slowdowns occurring due to Shared Memory feature they introduced some while ago...
https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/11062
https://nvidia.custhelp.com/app/answers/detail/a_id/5490
This was a while back, but I found out today. I believe a lot of people are still on older drivers due to this issue...
GitHub
Update (2023-10-31) This issue should now be entirely resolved. NVIDIA has made a help article to disable the system memory fallback behavior. Please upgrade to the latest driver (546.01) and follo...
Bunder 3D application settings (with the new driver) find your python.exe and apply the following setting
V2 hardware performance questions:
- I recently upgraded from an 2060 to a 3090 (24GB VRAM) - is there anything I’m supposed to do to make sure A1111 and ComfyUI go as fast as possible (maybe I need to reinstall if they do some kind of optimizations during install)
- I have 16GB RAM - will this be a bottleneck of some kind?
X- Disable "driver fallback" option on nVidia panel (if you get OUt of memory CUDA error, enable it again) ////// get rid of "--lowvram" command
VWhen using accelerate and sdxl_train.py from the commandline, instead of using the Kohya GUI, how do you specify the "epoch" or "max train epoch" parameters? I don't see any commandline argument being provided that seems related to these, and I can't seem to find any reference to this by googling
MHello lads! It's always me, an error generator. I was following this video https://www.youtube.com/watch?v=16-b1AjvyBE&t=1883s, and everything was fine, until you get to the calculation of the steps. Everything coincides except the last digit to be multiplied. For him it's 1300 / 1 / 1 * 1 * 2 while for me instead of 2, there's always 1. Solutions?
Gsame issues
SThe 2 for him is because he is using regularization images. If you are not it’s okay to not get the x2
MOkay, thank you!
SEpoch is listed as “—lr_scheduler_num_cycles=“x””
SNot sure about max but you can find this by setting it to a number that doesn’t occur elsewhere in the command line and then print the command and you’ll see the number you set it to
Vanyone know the outcome of triton on windows? I know it released recently and was undergoing testing and implementation.
VAwesome, thank you!
MSorry guys, I was trying to train a checkpoint using a custom model (GHArt/Realistic_Stock_Photo_V1.0_xl_fp16), but it gives me the following error. Can you help me, please? Thank you!
SHello... Might not be the right place to post this, but if I have 160 images of a character, what epoch and repeats is the best place to start for a lora?(using kohya)
EHey there community.
A quick random question for you. Besides for perhaps training SDXL via dreambooth, have you ever seen a need for more than 24gb vram gpu in your projects or adventures? I’m trying to cut down cloud computing to a minimum so seeing what’s logical and what isn’t to buy.
A quick random question for you. Besides for perhaps training SDXL via dreambooth, have you ever seen a need for more than 24gb vram gpu in your projects or adventures? I’m trying to cut down cloud computing to a minimum so seeing what’s logical and what isn’t to buy.
1Boy... Training on DB with a 3060 12GB VRAM works with your "best" settings, but it takes 8 hours for 10 instance pics and 2000 ref pics. Holy sh...... But i tried with your "12GB VRAM" suggestions that take only about 1 hour, and the results are....meehhhhhh.
1Looking forward to the results after the 8 hours
P
Fmore info needed
Fye please DM me
FNV link is different stuff
Fwe are not combining VRAM of them
Fwe are batching
Fso still multi GPU training is made
Fyou can download and upload as data
Fthen import next time