Software Engineering Courses (SECourses)•3y ago

My lora training just started .. roughly 4.5 hours. @Dr. Furkan Gözükara The old guide seems to be l

My lora training just started .. roughly 4.5 hours.
@Dr. Furkan Gözükara The old guide seems to be limiting with ram availability , but i think it can be slightly updated since kaggle has increased the ram. I didnt add -lowram argument

DR.SivaOP•11/26/23, 9:41 AM

gradient checkpoint as well

DDR.Siva My lora training just started .. roughly 4.5 hours. @Dr. Furkan Gözükara The old...

Furkan Gözükara SECourses•11/26/23, 9:50 AM

you dont need --lowvram anymore accurate

DDR.Siva My lora training just started .. roughly 4.5 hours. @Dr. Furkan Gözükara The old...

Furkan Gözükara SECourses•11/26/23, 9:51 AM

5200 steps will be actually 10400 step so it may cook. since uses dual gpu - hence batch count is being 2

DR.SivaOP•11/26/23, 9:52 AM

i made a mistake

DR.SivaOP•11/26/23, 9:53 AM

i didnt copy the training command ,i stoped and deleted the regu folder and started the next code in notebnook which had your settings

DR.SivaOP•11/26/23, 9:53 AM

i am starting the notebook afresh

Baran•11/26/23, 9:54 AM

@Dr. Furkan Gözükara can you explain why you are using a vae in your latest sdxl dreambooth training config?

Baran•11/26/23, 9:54 AM

"vae": "stabilityai/sdxl-vae"

DR.SivaOP•11/26/23, 10:03 AM

also Jpeg / PNG , i saw in one of your comment that to convert jpeg to PNG in case there are errors . training better with jpeg or png ?

DR.SivaOP•11/26/23, 10:09 AM

Also since kaggle gives 20Gb , for lora training , do we need to stop and deleted the regul folder once its gets copied into results ? this step can be skipped , so that we dont mess with the lora training setting ?

DR.SivaOP•11/26/23, 10:30 AM

I am getting error when trying to paste the training command
accelerate launch --num_cpu_threads_per_process=4 "./sdxl_train_network.py" --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --train_data_dir="/kaggle/working/results/img" --reg_data_dir="/kaggle/working/results/reg" --resolution="1024,1024" --output_dir="/kaggle/working/results/model" --logging_dir="/kaggle/working/results/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=64 --output_name="Lora_Gowthu" --lr_scheduler_num_cycles="8" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="7200" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01 --max_data_loader_n_workers="0" --bucket_reso_steps=64 --save_every_n_steps="1300" --mem_eff_attn --gradient_checkpointing --full_fp16 --xformers --bucket_no_upscale --noise_offset=0.0 --max_grad_norm=0.0 --no_half_vae --train_text_encoder --vae="stabilityai/sdxl-vae"

DR.SivaOP•11/26/23, 10:40 AM

2023-11-26 10:40:23.363084: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-26 10:40:23.363138: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-26 10:40:23.363201: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-26 10:40:23.366439: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-26 10:40:23.366486: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-26 10:40:23.366538: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
usage: sdxl_train_network.py [-h] [--v2] [--v_parameterization]
[--pretrained_model_name_or_path PRETRAINED_MODEL_NAME_OR_PATH]

DR.SivaOP•11/26/23, 10:42 AM

DR.SivaOP•11/26/23, 10:42 AM

am i doing something wrong ?

DR.SivaOP•11/26/23, 11:44 AM

for some weird reason it worked when i used your setting , but now its failing ..

DR.SivaOP•11/26/23, 12:14 PM

~~i think its the kaggle_sdxl_dreambooth_best.json which is getting downloaded each time~~

DR.SivaOP•11/26/23, 12:51 PM

ok hitting "start training" within koayaa_gui seems to work. When we copy the code - stop the code- remove the folder - paster the code - start the code from kaggle ,doesnt seems to work

DR.SivaOP•11/26/23, 12:52 PM

sdxl_train_network.py [-h] [--v2] [--v_parameterization] this .py doesnt seem to get started when we execute the training command separately.

fortune•11/26/23, 12:57 PM

Hello, guys, Now I'm looking for someone who is experienced in .Net. (US, UK prefered, Fluent English Level, European or American). Please DM if anybody is interested in.

DR.SivaOP•11/26/23, 1:12 PM

got stuck at point and had to restart.. Have to reinstall fully ?

Is there a way to retain all the files/setups ? before i come to hit the train button ,gpu time of 1 hour is lost..

DR.SivaOP•11/26/23, 2:58 PM

atlast... 1st epoch finished

DR.SivaOP•11/26/23, 3:06 PM

Now i think why dreambooth training could have failed.. I think hitting train within the koyaa GUI works better on kaggle , rather than executing the same code from notebook. After the lora gets created , tomorrow will try once more to train on dreambooth.

samopopo•11/26/23, 4:01 PM

@Dr. Furkan Gözükara Hello will you create anytime soon Stable diffusion video generator tutorial? Or img to video?

FFurkan Gözükara SECourses hi you need to show cmd output

Ned Shoaei•11/26/23, 5:38 PM

hello, i get this error in my log

Ned Shoaei•11/26/23, 5:39 PM

i'm using xl model with xl controlnet

DR.SivaOP•11/26/23, 5:42 PM

its late night now , can i leave kaggle running and just keep browser and pc running ?

DR.SivaOP•11/26/23, 5:42 PM

its now 50% over with 7 epochs generated

DR.SivaOP•11/26/23, 5:42 PM

still 9 to go .. 3.3hrs remaining

DR.SivaOP•11/26/23, 5:42 PM

help

DDR.Siva i didnt copy the training command ,i stoped and deleted the regu folder and star...

Furkan Gözükara SECourses•11/26/23, 9:02 PM

now you can directly run from gui

Furkan Gözükara SECourses•11/26/23, 9:02 PM

no need to stop

BBaran @Dr. Furkan Gözükara can you explain why you are using a vae in your latest sdxl...

Furkan Gözükara SECourses•11/26/23, 9:03 PM

we are embedding very best vae into the model

Ssamopopo @Dr. Furkan Gözükara Hello will you create anytime soon Stable diffusion video g...

Furkan Gözükara SECourses•11/26/23, 9:03 PM

hopefully planning soon

NNed Shoaei hello, i get this error in my log

Furkan Gözükara SECourses•11/26/23, 9:04 PM

which automatic111 are you using?

Furkan Gözükara SECourses•11/26/23, 9:04 PM

if you are using dreambooth extension version it is an old version

DDR.Siva its late night now , can i leave kaggle running and just keep browser and pc run...

Furkan Gözükara SECourses•11/26/23, 9:04 PM

yes you can

Furkan Gözükara SECourses•11/26/23, 9:04 PM

but max 12 hour is per session i think

Furkan Gözükara SECourses•11/26/23, 9:04 PM

just leave kaggle window open

Furkan Gözükara SECourses•11/26/23, 9:04 PM

you cna stop training and download checkpoints

Furkan Gözükara SECourses•11/26/23, 9:04 PM

before going to sleep

DR.SivaOP•11/26/23, 9:30 PM

Set alarm, just woke up, it's 3am now.. Downloaded all loras. Tom will do xyz of all loras, and see which is good.
Will give dreambooth training one more try. Have only 10 hours of gpu time.
Once again thanks a lot @Dr. Furkan Gözükara .thanks for replying for each of our messages..

DR.SivaOP•11/26/23, 9:31 PM

Gn guys

Digital [Starburst]•11/26/23, 11:24 PM

I learned something about using ControlNet with SDXL. Not every option in ControlNet has an SDXL model. I suspect that is why some say that ControlNet is only partially compatible with SDXL.

DDigital [Starburst]I learned something about using ControlNet with SDXL. Not every option in Contr...

Furkan Gözükara SECourses•11/26/23, 11:44 PM

ah i see

Furkan Gözükara SECourses•11/26/23, 11:44 PM

accurate

Furkan Gözükara SECourses•11/26/23, 11:44 PM

you need to have XL in the name of the selected model

Furkan Gözükara SECourses•11/26/23, 11:44 PM

class images datasets updated : https://civitai.com/articles/2285/massive-4k-resolution-woman-and-man-class-ground-truth-stable-diffusion-regularization-images-dataset

Massive 4K Resolution Woman & Man Class Ground Truth Stable Diffusi...

Download link > https://www.patreon.com/posts/87700469 The best ever released Stable Diffusion classification / regularization images dataset ju...

Zono50•11/27/23, 12:32 AM

What's the best way to upscale human photos to get the best detail?

FFurkan Gözükara SECourses you cna stop training and download checkpoints

DR.SivaOP•11/27/23, 2:09 AM

Just need a clarification, if we stop training midway, can we continue again?
Also how can we prevent installing from the start for each notebook we create on kaggle or any platform?
In case we can preserve the setup, we can simply go and change, say, the training images, and simply hit train.. Last time on runpod, it was consuming space, so I was loosing money.
I

DR.SivaOP•11/27/23, 3:13 AM

For those who get Automatic1111 stuck in cmd , here is the solution : https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/2452

TLDR : Just go to properties of CMD and uncheck "Quick Edit Mode"

GitHub

SD command window is just stucked, need to hit Enter to continue · ...

No error when this happen In Windows, i noticed some recently version have this problem, when i set hundred to thousand batch count, it just stucked random at batch xx (sometime at batch 2, sometim...

My lora training just started .. roughly 4.5 hours. @Dr. Furkan Gözükara The old guide seems to be l

Similar Threads