Software Engineering Courses (SECourses)•3y ago

kaggle error : File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.p

kaggle error :

File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=====================================================
./sdxl_train.py FAILED
-----------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
-----------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-11-25_17:18:32
host : 5d9e43f0dfc2
rank : 1 (local_rank: 1)
exitcode : -6 (pid: 1262)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 1262

Aldo Raine•11/25/23, 6:09 PM

I am getting the same thing using the patreon v12 file:

[E ProcessGroupNCCL.cpp:460] To avoid data inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
what(): [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804412 milliseconds before timing out.
93%|████████████████████████████████████▎ | 1503/1616 [30:08<02:16, 1.21s/it]Traceback (most recent call last):
File "/opt/conda/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=====================================================
./sdxl_train.py FAILED
-----------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
-----------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-11-25_18:01:51
host : 10ed691951a5
rank : 1 (local_rank: 1)
exitcode : -6 (pid: 1077)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 1077
=====================================================

T h e A n a y•11/25/23, 8:24 PM

T h e A n a y•11/25/23, 8:25 PM

anyone encountering the same error nd has a fix?

7740 hi guys! is animal lora training is the same as for humans? and what about regul...

Furkan Gözükara SECourses•11/25/23, 8:42 PM

same but i dont have reg images

Rrafraf In the video you show these aspect ratios on white screen, but the official stab...

Furkan Gözükara SECourses•11/25/23, 8:42 PM

official res is better

Furkan Gözükara SECourses•11/25/23, 8:42 PM

i prepared those as well

Furkan Gözükara SECourses•11/25/23, 8:42 PM

man dataset already on patreon

Furkan Gözükara SECourses•11/25/23, 8:42 PM

but i am preparing woman too

Furkan Gözükara SECourses•11/25/23, 8:42 PM

today i will upload

Furkan Gözükara SECourses•11/25/23, 8:44 PM

AAIGambino When having adetailer on, how can I make it so its only bringing detail of my gu...

Furkan Gözükara SECourses•11/25/23, 8:44 PM

yes

Furkan Gözükara SECourses•11/25/23, 8:44 PM

in last video i have shown this

Furkan Gözükara SECourses•11/25/23, 8:44 PM

watch a detaler part

AAldo Raine I am getting the same thing using the patreon v12 file: [E ProcessGroupNCCL.cpp...

Furkan Gözükara SECourses•11/25/23, 8:45 PM

what is your resolutio0n

TT h e A n a y Click to see attachment

Furkan Gözükara SECourses•11/25/23, 8:45 PM

this is install error

Furkan Gözükara SECourses•11/25/23, 8:45 PM

try to reinstall

Furkan Gözükara SECourses•11/25/23, 8:45 PM

python 3.10.11 use it

Furkan Gözükara SECourses•11/25/23, 8:46 PM

you can also manually instlal bitsandbytes

FFurkan Gözükara SECourses this is install error

T h e A n a y•11/25/23, 8:52 PM

i tried reinstalling it didnt work

T h e A n a y•11/25/23, 8:52 PM

also tried google colab

T h e A n a y•11/25/23, 8:53 PM

FFurkan Gözükara SECourses you can also manually instlal bitsandbytes

T h e A n a y•11/25/23, 8:53 PM

i dont really know how to do that

T h e A n a y•11/25/23, 8:53 PM

there are a lot of ppl on kohya ss issues page with the same issue

T h e A n a y•11/25/23, 8:56 PM

@Dr. Furkan Gözükara do u know any other programs for lora training

FFurkan Gözükara SECourses what is your resolutio0n

Aldo Raine•11/25/23, 8:58 PM

1024x1024

Digital [Starburst]•11/25/23, 9:09 PM

I'm getting an error with TensorRT when trying to install Dreambooth

Digital [Starburst]•11/25/23, 9:13 PM

Um...to run Tensor on low VRAM do you need to count on needing a liquid cooled GPU?

Digital [Starburst]•11/25/23, 9:14 PM

How hot will she run?

Digital [Starburst]•11/25/23, 9:15 PM

*** Failed reading extension data from Git repository (se_dreambooth_extension)

DDigital [Starburst]Um...to run Tensor on low VRAM do you need to count on needing a liquid cooled G...

T h e A n a y•11/25/23, 9:15 PM

u dont really need liquid cooling if ur gpu temps are normal acc to the gpu u are running

Digital [Starburst]•11/25/23, 9:16 PM

TypeError: HEAD is a detached symbolic reference as it points to 'dc413a14379b165355502d9f65856c40a4bb5b6f'

virginia•11/25/23, 9:21 PM

hello Furkan, checkpoint training settings for kaggle done?

TT h e A n a y @Dr. Furkan Gözükara do u know any other programs for lora training

Furkan Gözükara SECourses•11/25/23, 9:29 PM

kohya is best

Furkan Gözükara SECourses•11/25/23, 9:29 PM

i would try to solve issue

Furkan Gözükara SECourses•11/25/23, 9:29 PM

it is easy to solve

Furkan Gözükara SECourses•11/25/23, 9:29 PM

are you my patreon supporteR?

AAldo Raine 1024x1024

Furkan Gözükara SECourses•11/25/23, 9:29 PM

you shouldnt get any error

Furkan Gözükara SECourses•11/25/23, 9:30 PM

run script again

Furkan Gözükara SECourses•11/25/23, 9:30 PM

i think you had error while caching

DDigital [Starburst]I'm getting an error with TensorRT when trying to install Dreambooth

Furkan Gözükara SECourses•11/25/23, 9:30 PM

tensorRT and dreambooth extension of automatic1111 version wont work

DDigital [Starburst]TypeError: HEAD is a detached symbolic reference as it points to 'dc413a14379b16...

Furkan Gözükara SECourses•11/25/23, 9:31 PM

you can ignore it

Furkan Gözükara SECourses•11/25/23, 9:31 PM

woprks grewat

Vvirginia hello Furkan, checkpoint training settings for kaggle done?

Furkan Gözükara SECourses•11/25/23, 9:31 PM

yes

Furkan Gözükara SECourses•11/25/23, 9:31 PM

https://youtu.be/16-b1AjvyBE

YouTubeSECourses

How To Do Stable Diffusion XL (SDXL) DreamBooth Training For Free -...

Master Stable Diffusion XL Training on Kaggle for Free!

Welcome to this comprehensive tutorial where I'll be guiding you through the exciting world of setting up and training Stable Diffusion XL (SDXL) with Kohya on a free Kaggle account. This video is your one-stop resource for learning everything from initiating a Kaggle session with dual ...