I get 1.2s/it on my 4090, that is why I was wondering why it's slower on a p100
I get 1.2s/it on my 4090, that is why I was wondering why it's slower on a p100
JOh wait, p100 isn't that new. Nevermind me.
JI was thinking about the A100
MWith kaggle's double gpu it's around 1.3s/it but it has only 14GB Vram
L@Dr. Furkan Gözükara have you tried multi gpu lora training with kohya?
ddp_bucket_view and the other parameter dont seem to exist even on dev branchL
BWhats the best tool for scanning a large dataset folder for duplicate images? (Some duplicates might be different size / res)
VSo I'm using all the latest instructions and settings, and I'm running training on a 4090 in Runpod. When I watch the video, Dr Furkan gets a total training time of 50m-2h using a 3090 on Runpod. For some reason I'm getting a training time of ~3hours, despite being on a 4090.
Did something change about the training between the video publication and now, making it more intensive? Or am I doing something wrong?
Did something change about the training between the video publication and now, making it more intensive? Or am I doing something wrong?
VThis is with 15 example images, scaled to 1024x1024, and using the reg set of men at 1024x1024 as well.
PIn the vast expanse of the whimsical world of Widdendream, where every creature was as unique as a snowflake in a winter flurry, there stood two friends: little Lyla Longbeak and the towering Tara Tattercloak. Lyla, with her inquisitive eyes and a beak sharp enough to pick the seeds of wisdom from the fruits of knowledge, gazed up in awe at her friend. Tara, draped in a cloak of midnight blue, tattered at the edges from embracing every thorn and rose life offered, towered above like a gentle giant, her eyes kind pools of understanding.
Together, they were a testament to the beauty of difference, a duo that danced in harmony despite their contrasting tunes. The room they shared was filled with oddities and ends, with hanging cages that held not birds but blooming ideas, and a television that was old and wise, flickering with the stories of yore.
"See, Lyla," Tara's voice rustled like the leaves of an ancient tree, "we are like these cages, different in size and shape, yet home to ideas just as bright and beautiful." Lyla nodded, her heart swelling with the warmth of acceptance.
Their friendship was a mosaic, a splendid tapestry woven from threads of myriad textures and hues, teaching all who visited Widdendream that diversity was not just to be tolerated, but celebrated, for it was the very essence of life's enchanting tapestry.
Together, they were a testament to the beauty of difference, a duo that danced in harmony despite their contrasting tunes. The room they shared was filled with oddities and ends, with hanging cages that held not birds but blooming ideas, and a television that was old and wise, flickering with the stories of yore.
"See, Lyla," Tara's voice rustled like the leaves of an ancient tree, "we are like these cages, different in size and shape, yet home to ideas just as bright and beautiful." Lyla nodded, her heart swelling with the warmth of acceptance.
Their friendship was a mosaic, a splendid tapestry woven from threads of myriad textures and hues, teaching all who visited Widdendream that diversity was not just to be tolerated, but celebrated, for it was the very essence of life's enchanting tapestry.
Mthis is typical of my 4090. He may have used xformers which I do not since I want higher quality
VGotcha, thanks for the reply!
ALooking for the best workflow for a apparell mockup task:
I have a t-shirt design in transparent png and want to insert it onto a person wearing a plain t-shirt. The person is in a dynamic pose, so I need to get the shading, wrinkles done right.
Hope you can give me advice on how to approach this
I have a t-shirt design in transparent png and want to insert it onto a person wearing a plain t-shirt. The person is in a dynamic pose, so I need to get the shading, wrinkles done right.
Hope you can give me advice on how to approach this
SAnyone messed around with training TI embeddings? Trying to figure out why my TI + LoRA results are coming out with artifacts and over saturation. Seems like it could be a scheduler issue or VAE, but not sure right now.
VIs there a general rule for how well dreambooth training scales with multi-gpu? I used Dr Furkan's guide on a single 4090 and it took about 3 hours, decided to try it with a 6x 4090 setup, spawned it from cli with
... and the training only went down to 2 1/2 hours.
... and the training only went down to 2 1/2 hours.
Syou need to increase the batch size when doing multi gpu
Sie. increase the number of images processed at a time
VThanks, testing it now with a batch size of 6. I googled that earlier and the top comment I found said you want to keep a small batch size for multi gpu in order to make sure each GPU is always fully utilized, confusing
VHmm, so setting to anything higher than 2 leads to it crashing due to out of memory errors. However, with a batch size of 2 it went to a 4 hour estimate, whereas it was at 2.5 hours @ batch size of 1.
VHmm, could this be related? It seems like with 6 processes, batch size of 2 and multi-gpu enabled, I get 96 epochs and 100 batches per epoch. But with 1 process, batch size of 1 and multi-gpu disabled, I get 8 epochs @ 1200 batches per epoch.
VGPUs are being heavily utilized with 6 processes, batch size 2, multi-gpu. So it's not like they aren't doing anything. And yet, it's gonna take 4 hours and 15 minutes like this. Whereas with a single GPU it takes 3 hours.
V1.6s/it with 6 GPU, 1.07s/it with 1 GPU.
Fok time to check messages
Fhello welcome
Fcan you explain me more info
Fwhat you are trying to do
Fovertrained
MHello guys. I'm new to the discord and wanted to ask for information. I'm having a big problem these days, trying to train the lora sdxl on colab. Do you know who could give me a hand?
MThanks in advice 
F1.2 s / it is great
Fwelcome
Fsadly i have no colab for sdxl
Fbut we have automatic1111 web ui for kaggle
Fsupports all controlnet and sdxl
Fi did multi sdxl dreambooth and lora training on kaggle
Fhe is making some fixes to multi gpu recently
Mhi furkan! Thanks for answering me I also recently tried to use kaggle, but unfortunately I saw that they blocked it and it no longer works. Is there a way to bypass it? Thank you
I also recently tried to use kaggle, but unfortunately I saw that they blocked it and it no longer works. Is there a way to bypass it? Thank youLso how did you get it to actually offload across multiple gpus? I tried accelerate config, accelerate arguments etc
Fthere is bypass
Fon kaggle
Fit auto does that on kaggle
Fnow on runpod as well
FMSorry for asking but I've been going crazy for a week. Could you tell me how to solve it please?
Lwell I'm on local hardware haha, would've loved to know what runpod and kaggle do, that just scales across GPUs, though they might be just in SLI/nvlink
Fvector based search
Fi have script for this let me share on patreon
Bok amazing thankyou
Fare you trying on kaggle