I have used the CN py from patreon to update: python CND_runpod.py still having issues
I have used the CN py from patreon to update:
python CND_runpod.py
still having issues
python CND_runpod.py
still having issues
Nthis is my extension tab:
DPastebin
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Dfor time being will train new image batch with lora. Last time there were too many repeating images which made lora inherit certain aspect of dress/color into it and was making output with repative features of the images
Fjust run cell again
Fload and hit again
Fit will cache the remaining images
Fwhat(): [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802130 milliseconds before timing out.
Fcaching taking too long thus failing
For reduce repeating count
Flike half it and double the epoch
Fhi you need to show cmd output
Fdid you kill previous instance and started yourself?
FFor 4090 definitely do SDXL DreamBooth with our config from here on Kohya
FYouTubeSECourses
Patreon post for config files 
https://www.patreon.com/posts/89213064
SECourses Discord To Get Full Support
https://discord.com/servers/software-engineering-courses-secourses-772774097734074388
My LinkedIn
https://www.linkedin.com/in/furkangozukara/
My Twitter
https://twitter.com/GozukaraFurkan
My Instagram
https://www.instagra...
https://www.patreon.com/posts/89213064
SECourses Discord To Get Full Support
https://discord.com/servers/software-engineering-courses-secourses-772774097734074388
My LinkedIn
https://www.linkedin.com/in/furkangozukara/
My Twitter
https://twitter.com/GozukaraFurkan
My Instagram
https://www.instagra...
FF24GB_TextEncoder.json
Fthat extension is not maintained anymore.
FGitHub
When caching taking over 30 minutes it time outs. I wonder if this is Kohya related The message is from Kaggle notebook caching latents. checking cache validity... 100%|████████████████████████████...
DThanks a lot for taking time and effort to resolve our issues.. Thanks again
Dcurrently doing lora training using your guide
D
DMy lora training just started .. roughly 4.5 hours.
@Dr. Furkan Gözükara The old guide seems to be limiting with ram availability , but i think it can be slightly updated since kaggle has increased the ram. I didnt add -lowram argument
@Dr. Furkan Gözükara The old guide seems to be limiting with ram availability , but i think it can be slightly updated since kaggle has increased the ram. I didnt add -lowram argument
Dgradient checkpoint as well
Fyou dont need --lowvram anymore accurate
F5200 steps will be actually 10400 step so it may cook. since uses dual gpu - hence batch count is being 2
Di made a mistake
Di didnt copy the training command ,i stoped and deleted the regu folder and started the next code in notebnook which had your settings
Di am starting the notebook afresh
B@Dr. Furkan Gözükara can you explain why you are using a vae in your latest sdxl dreambooth training config?
B"vae": "stabilityai/sdxl-vae"
Dalso Jpeg / PNG , i saw in one of your comment that to convert jpeg to PNG in case there are errors . training better with jpeg or png ?
DAlso since kaggle gives 20Gb , for lora training , do we need to stop and deleted the regul folder once its gets copied into results ? this step can be skipped , so that we dont mess with the lora training setting ?
DI am getting error when trying to paste the training command
accelerate launch --num_cpu_threads_per_process=4 "./sdxl_train_network.py" --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --train_data_dir="/kaggle/working/results/img" --reg_data_dir="/kaggle/working/results/reg" --resolution="1024,1024" --output_dir="/kaggle/working/results/model" --logging_dir="/kaggle/working/results/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=64 --output_name="Lora_Gowthu" --lr_scheduler_num_cycles="8" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="7200" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01 --max_data_loader_n_workers="0" --bucket_reso_steps=64 --save_every_n_steps="1300" --mem_eff_attn --gradient_checkpointing --full_fp16 --xformers --bucket_no_upscale --noise_offset=0.0 --max_grad_norm=0.0 --no_half_vae --train_text_encoder --vae="stabilityai/sdxl-vae"
accelerate launch --num_cpu_threads_per_process=4 "./sdxl_train_network.py" --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --train_data_dir="/kaggle/working/results/img" --reg_data_dir="/kaggle/working/results/reg" --resolution="1024,1024" --output_dir="/kaggle/working/results/model" --logging_dir="/kaggle/working/results/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=64 --output_name="Lora_Gowthu" --lr_scheduler_num_cycles="8" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="7200" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01 --max_data_loader_n_workers="0" --bucket_reso_steps=64 --save_every_n_steps="1300" --mem_eff_attn --gradient_checkpointing --full_fp16 --xformers --bucket_no_upscale --noise_offset=0.0 --max_grad_norm=0.0 --no_half_vae --train_text_encoder --vae="stabilityai/sdxl-vae"
D2023-11-26 10:40:23.363084: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-26 10:40:23.363138: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-26 10:40:23.363201: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-26 10:40:23.366439: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-26 10:40:23.366486: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-26 10:40:23.366538: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
usage: sdxl_train_network.py [-h] [--v2] [--v_parameterization]
[--pretrained_model_name_or_path PRETRAINED_MODEL_NAME_OR_PATH]
2023-11-26 10:40:23.363138: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-26 10:40:23.363201: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-26 10:40:23.366439: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-26 10:40:23.366486: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-26 10:40:23.366538: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
/opt/conda/lib/python3.10/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
usage: sdxl_train_network.py [-h] [--v2] [--v_parameterization]
[--pretrained_model_name_or_path PRETRAINED_MODEL_NAME_OR_PATH]
D
Dam i doing something wrong ?
Dfor some weird reason it worked when i used your setting , but now its failing .. 
DDok hitting "start training" within koayaa_gui seems to work. When we copy the code - stop the code- remove the folder - paster the code - start the code from kaggle ,doesnt seems to work
Dsdxl_train_network.py [-h] [--v2] [--v_parameterization] this .py doesnt seem to get started when we execute the training command separately.
FHello, guys, Now I'm looking for someone who is experienced in .Net. (US, UK prefered, Fluent English Level, European or American). Please DM if anybody is interested in.
Dgot stuck at point and had to restart.. Have to reinstall fully ? Is there a way to retain all the files/setups ? before i come to hit the train button ,gpu time of 1 hour is lost..
Is there a way to retain all the files/setups ? before i come to hit the train button ,gpu time of 1 hour is lost.. Datlast... 1st epoch finished
DNow i think why dreambooth training could have failed.. I think hitting train within the koyaa GUI works better on kaggle , rather than executing the same code from notebook. After the lora gets created , tomorrow will try once more to train on dreambooth.
S@Dr. Furkan Gözükara Hello will you create anytime soon Stable diffusion video generator tutorial? Or img to video?
Nhello, i get this error in my log
Ni'm using xl model with xl controlnet
Dits late night now , can i leave kaggle running and just keep browser and pc running ?
Dits now 50% over with 7 epochs generated
Dstill 9 to go .. 3.3hrs remaining