Anyone messed around with training TI embeddings? Trying to figure out why my TI + LoRA results are

V

Vexmachina•12/22/23, 10:54 PM

Is there a general rule for how well dreambooth training scales with multi-gpu? I used Dr Furkan's guide on a single 4090 and it took about 3 hours, decided to try it with a 6x 4090 setup, spawned it from cli with

accelerate launch --num_cpu_threads_per_process=4 --num_processes=6 --multi_gpu --num_machines=1 --gpu_ids=0,1,2,3,4,5 "./sdxl_train.py"

accelerate launch --num_cpu_threads_per_process=4 --num_processes=6 --multi_gpu --num_machines=1 --gpu_ids=0,1,2,3,4,5 "./sdxl_train.py"

accelerate launch --num_cpu_threads_per_process=4 --num_processes=6 --multi_gpu --num_machines=1 --gpu_ids=0,1,2,3,4,5 "./sdxl_train.py"

accelerate launch --num_cpu_threads_per_process=4 --num_processes=6 --multi_gpu --num_machines=1 --gpu_ids=0,1,2,3,4,5 "./sdxl_train.py"

... and the training only went down to 2 1/2 hours.

S

stillmotionOP•12/22/23, 10:58 PM

you need to increase the batch size when doing multi gpu

S

stillmotionOP•12/22/23, 10:58 PM

ie. increase the number of images processed at a time

V

Vexmachina•12/22/23, 11:06 PM

Thanks, testing it now with a batch size of 6. I googled that earlier and the top comment I found said you want to keep a small batch size for multi gpu in order to make sure each GPU is always fully utilized, confusing

V

Vexmachina•12/22/23, 11:14 PM

Hmm, so setting

--train_batch_size="2"

--train_batch_size="2"

--train_batch_size="2"

--train_batch_size="2"

to anything higher than 2 leads to it crashing due to out of memory errors. However, with a batch size of 2 it went to a 4 hour estimate, whereas it was at 2.5 hours @ batch size of 1.

V

Vexmachina•12/22/23, 11:37 PM

Hmm, could this be related? It seems like with 6 processes, batch size of 2 and multi-gpu enabled, I get 96 epochs and 100 batches per epoch. But with 1 process, batch size of 1 and multi-gpu disabled, I get 8 epochs @ 1200 batches per epoch.

num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 100
  num epochs / epoch数: 96
  batch size per device / バッチサイズ: 2
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 100
  num epochs / epoch数: 96
  batch size per device / バッチサイズ: 2
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 100
  num epochs / epoch数: 96
  batch size per device / バッチサイズ: 2
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 100
  num epochs / epoch数: 96
  batch size per device / バッチサイズ: 2
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

running training / 学習開始
  num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 1200
  num epochs / epoch数: 8
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

running training / 学習開始
  num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 1200
  num epochs / epoch数: 8
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

running training / 学習開始
  num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 1200
  num epochs / epoch数: 8
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

running training / 学習開始
  num examples / サンプル数: 600
  num batches per epoch / 1epochのバッチ数: 1200
  num epochs / epoch数: 8
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 9600

V

Vexmachina•12/22/23, 11:42 PM

GPUs are being heavily utilized with 6 processes, batch size 2, multi-gpu. So it's not like they aren't doing anything. And yet, it's gonna take 4 hours and 15 minutes like this. Whereas with a single GPU it takes 3 hours.

V

Vexmachina•12/22/23, 11:42 PM

1.6s/it with 6 GPU, 1.07s/it with 1 GPU.

F

Furkan Gözükara SECourses•12/22/23, 11:48 PM

ok time to check messages

DDarth Teethius Hello, new to the patreon, looking at the config files and i notice i can't find...

F

Furkan Gözükara SECourses•12/22/23, 11:49 PM

hello welcome

F

Furkan Gözükara SECourses•12/22/23, 11:49 PM

can you explain me more info

F

Furkan Gözükara SECourses•12/22/23, 11:49 PM

what you are trying to do

JJozsef The checkpoint that I have thought a style with, now produces that style even wi...

F

Furkan Gözükara SECourses•12/22/23, 11:49 PM

overtrained

M

Max•12/22/23, 11:51 PM

Hello guys. I'm new to the discord and wanted to ask for information. I'm having a big problem these days, trying to train the lora sdxl on colab. Do you know who could give me a hand?

M

Max•12/22/23, 11:51 PM

Thanks in advice

MMetaphysix I used your crop & resizer tool and made them 896x1152, It's still using 14.9 GB...

F

Furkan Gözükara SECourses•12/22/23, 11:52 PM

great

JJozsef I get 1.2s/it on my 4090, that is why I was wondering why it's slower on a p100

F

Furkan Gözükara SECourses•12/22/23, 11:53 PM

1.2 s / it is great

MMax Hello guys. I'm new to the discord and wanted to ask for information. I'm having...

F

Furkan Gözükara SECourses•12/22/23, 11:53 PM

welcome

F

Furkan Gözükara SECourses•12/22/23, 11:53 PM

sadly i have no colab for sdxl

F

Furkan Gözükara SECourses•12/22/23, 11:53 PM

but we have automatic1111 web ui for kaggle

F

Furkan Gözükara SECourses•12/22/23, 11:53 PM

supports all controlnet and sdxl

Llucy @Dr. Furkan Gözükara have you tried multi gpu lora training with kohya? `ddp_buc...

F

Furkan Gözükara SECourses•12/22/23, 11:54 PM

i did multi sdxl dreambooth and lora training on kaggle

F

Furkan Gözükara SECourses•12/22/23, 11:54 PM

he is making some fixes to multi gpu recently

FFurkan Gözükara SECourses supports all controlnet and sdxl

M

Max•12/22/23, 11:55 PM

hi furkan! Thanks for answering me

I also recently tried to use kaggle, but unfortunately I saw that they blocked it and it no longer works. Is there a way to bypass it? Thank you

FFurkan Gözükara SECourses i did multi sdxl dreambooth and lora training on kaggle

L

lucy•12/22/23, 11:56 PM

so how did you get it to actually offload across multiple gpus? I tried accelerate config, accelerate arguments etc

MMax hi furkan! Thanks for answering me 🙂 I also recently tried to use kaggle, but u...

F

Furkan Gözükara SECourses•12/23/23, 12:04 AM

yes

F

Furkan Gözükara SECourses•12/23/23, 12:04 AM

there is bypass

F

Furkan Gözükara SECourses•12/23/23, 12:04 AM

on kaggle

Llucy so how did you get it to actually offload across multiple gpus? I tried accelera...

F

Furkan Gözükara SECourses•12/23/23, 12:04 AM

it auto does that on kaggle

F

Furkan Gözükara SECourses•12/23/23, 12:04 AM

now on runpod as well

F

Furkan Gözükara SECourses•12/23/23, 12:04 AM

our kaggle notebooks : https://www.patreon.com/collection/258108?view=expanded

FFurkan Gözükara SECourses on kaggle

M

Max•12/23/23, 12:05 AM

Sorry for asking but I've been going crazy for a week. Could you tell me how to solve it please?

L

lucy•12/23/23, 12:05 AM

well I'm on local hardware haha, would've loved to know what runpod and kaggle do, that just scales across GPUs, though they might be just in SLI/nvlink

BB0nd Whats the best tool for scanning a large dataset folder for duplicate images? (S...

F

Furkan Gözükara SECourses•12/23/23, 12:22 AM

vector based search

F

Furkan Gözükara SECourses•12/23/23, 12:22 AM

i have script for this let me share on patreon

B

B0nd•12/23/23, 12:22 AM

ok amazing thankyou

Llucy well I'm on local hardware haha, would've loved to know what runpod and kaggle d...

F