Software Engineering Courses (SECourses)•3y ago

1024 sdxl fail again ------------------------------------------------------------ Failures: [1]:

1024 sdxl fail again ------------------------------------------------------------
Failures:
[1]:
time : 2023-10-10_15:03:15
host : 19fe5077aa6e
rank : 1 (local_rank: 1)
exitcode : 2 (pid: 1098)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-10-10_15:03:15
host : 19fe5077aa6e
rank : 0 (local_rank: 0)
exitcode : 2 (pid: 1097)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

virginiaOP•10/10/23, 3:04 PM

train_network.py: error: argument --max_data_loader_n_workers: expected one argument
train_network.py: error: argument --max_data_loader_n_workers: expected one argument
Traceback (most recent call last):
File "/opt/conda/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Zono50•10/10/23, 3:18 PM

trying to put my face on this image, but it's not going well, tried regular lora with positive prompt, roop and auto detailer, but never gets the face well

Zono50•10/10/23, 3:18 PM

I assume it has to do with distance and darkness

aelvelis•10/10/23, 3:21 PM

Hi friends, I have this problem: Very slow image generation speed when using SDXL 1.0 on RTX 3060 Ti GPU and 16gb RAM. If i generate 1024x1024 with 20 steps then the generation time is 10 minutes, if for example 768x1024 then the generation speed is from 20 minutes. I have seen many people have generation speeds of only a couple minutes on the same GPU. I am using AUTOMATIC1111 version v1.6.0 and the start parameter --opt-sdp-attention. I don't get any errors about lack of VRAM, just a very slow generation speed. Maybe I should do a clean SD installation?

aelvelis•10/10/23, 3:22 PM

I enabled sdxl.vae and got a reduction in image generation time. But I got an error. Anyway it's not a 10 minutes wait anymore:)
When generating at 40 steps, the generation time doubles.

EElisoli so no bugs using 4090 with automatic1111\kohya ?

mtnmecca•10/10/23, 3:29 PM

none with A1111 so far and haven't tried kohya yet. Will in the next few days.

mtnmecca•10/10/23, 3:41 PM

What are other good rare tokens to use for training besides ohwx? I would like to train other subjects

Tthesilvermoth I also noticed the jump in quality. It has incredible details without hires fix

mtnmecca•10/10/23, 3:49 PM

I also just noticed the change in processing units. Never paid attention before but I was getting 1.6 sec/Iteration on 3060 and now I'm getting 0.31 sec/iteration (or 3.21 iterations/sec). 4-5x faster and better quality

aelvelis•10/10/23, 4:01 PM

I think I was able to fix the slow generation speed on the 3060 ti. I now generate a 768x1024 image in 30-40 seconds. The problem is with Nvidia drivers from version 531 and up. And also I used --xformers parameter instead of --opt-sdp-attention.
https://github.com/vladmandic/automatic/discussions/1285

GitHub

nVidia drivers change in memory management · vladmandic automatic ·...

it seems that nVidia changed memory management in the latest versions of drivers, specifically 532 and 535 new behavior is that once gpu vram is exhausted, it will actually use shared memory thus c...

LEGACIES•10/10/23, 4:53 PM

Hey guys
I'm a bit lost, can I use kohya to train model for SD 1.5?
If yes, which one gives better result between Dreambooth and Kohya?

Elisoli•10/10/23, 5:43 PM

A6000 training sdxl kohya - 48gb config file, around 8000 steps, 3 hours makes sense?

EElisoli A6000 training sdxl kohya - 48gb config file, around 8000 steps, 3 hours makes s...

mtnmecca•10/10/23, 5:48 PM

yes

EElisoli If you put of a survey of most wanted tutorials i think itll maybe easier for yo...

Furkan Gözükara SECourses•10/10/23, 5:56 PM

good idea

Mmayna Hello, Started to follow your course on youtube for quality studio images, when...

Furkan Gözükara SECourses•10/10/23, 5:56 PM

i see there dreambooth and lora tabs

Furkan Gözükara SECourses•10/10/23, 5:56 PM

DDeleted User Running on local URL: http://127.0.0.1:7860 To create a public link, set `shar...

Furkan Gözükara SECourses•10/10/23, 5:56 PM

you need special notebook we shared on patreon

Furkan Gözükara SECourses•10/10/23, 5:57 PM

https://www.patreon.com/posts/run-on-free-like-88714330

Furkan Gözükara SECourses•10/10/23, 5:57 PM

https://www.patreon.com/posts/kohya-sdxl-lora-88397937

Vvirginia 1024 sdxl fail again ----------------------------------------------------------...

Furkan Gözükara SECourses•10/10/23, 5:57 PM

you really are not following what i am telling you

Aaelvelis Hi friends, I have this problem: Very slow image generation speed when using SDX...

Furkan Gözükara SECourses•10/10/23, 5:58 PM

add --medvram

Furkan Gözükara SECourses•10/10/23, 5:58 PM

assuming it is 12 gb vram

LLEGACIES Hey guys I'm a bit lost, can I use kohya to train model for SD 1.5? If yes, whic...

Furkan Gözükara SECourses•10/10/23, 5:58 PM

yes you can use. but dreambooth extension is better for sd 1.5

Furkan Gözükara SECourses•10/10/23, 5:59 PM

i will search best parameters for sd 1.5 too for kohya

Furkan Gözükara SECourses•10/10/23, 5:59 PM

after sdxl dreambooth hopefully

LEGACIES•10/10/23, 6:00 PM

do you have best setting for dreambooth model SD 1,5? I have take a rtx3090 on runpod since I only have 12gb vram on my pc

LLEGACIES do you have best setting for dreambooth model SD 1,5? I have take a rtx3090 on r...

Furkan Gözükara SECourses•10/10/23, 6:02 PM

i dont have yet sadly for kohya

Furkan Gözükara SECourses•10/10/23, 6:02 PM

it is on my research list

Furkan Gözükara SECourses•10/10/23, 6:02 PM

but i have for extension

Furkan Gözükara SECourses•10/10/23, 6:03 PM

https://www.patreon.com/posts/best-settings-of-85192985

Furkan Gözükara SECourses•10/10/23, 6:03 PM

https://www.patreon.com/posts/runpod-auto-84716845

LEGACIES•10/10/23, 6:03 PM

thanks was looking for dreambooth

Furkan Gözükara SECourses•10/10/23, 6:04 PM

1 auto installer

Furkan Gözükara SECourses•10/10/23, 6:04 PM

2 settings

LEGACIES•10/10/23, 6:04 PM

Is it compulsory to use class images?

LLEGACIES Is it compulsory to use class images?

Furkan Gözükara SECourses•10/10/23, 6:04 PM

not but improves

Furkan Gözükara SECourses•10/10/23, 6:04 PM

i have class images too

Furkan Gözükara SECourses•10/10/23, 6:04 PM

https://www.patreon.com/posts/massive-4k-woman-87700469

FFurkan Gözükara SECourses https://www.patreon.com/posts/best-settings-of-85192985

LEGACIES•10/10/23, 6:04 PM

why is there 2 links?

LLEGACIES why is there 2 links?

Furkan Gözükara SECourses•10/10/23, 6:05 PM

1 settings

Furkan Gözükara SECourses•10/10/23, 6:05 PM

1 auto installer

Furkan Gözükara SECourses•10/10/23, 6:05 PM

read all links

LEGACIES•10/10/23, 6:05 PM

ah ok thank you

LEGACIES•10/10/23, 6:05 PM

will go through it

FFurkan Gözükara SECourses assuming it is 12 gb vram

aelvelis•10/10/23, 6:13 PM

Yes, I ran the --medvram-sdxl command and it really helped for my 3060 ti. The only downside is the long wait for the first image to be generated. Also xformers helped me because opt-sdp-attention gives extremely low generation speed for sdxl.

Aaelvelis Yes, I ran the --medvram-sdxl command and it really helped for my 3060 ti. The o...

Furkan Gözükara SECourses•10/10/23, 6:17 PM

ah u have ti

Furkan Gözükara SECourses•10/10/23, 6:17 PM

it has only 6 gb vram

Narva•10/10/23, 6:18 PM

: ERROR Error while calling W&B API: user is not logged in (<Response [401]>)
Problem at: /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/tracking.py 294 init
wandb: ERROR The API key is either invalid or missing, or the host is incorrect. To resolve this issue, you may try running the 'wandb login --host [hostname]' command. The host defaults to 'https://api.wandb.ai' if not specified. (Error 401: Unauthorized)
Traceback (most recent call last):
File "/workspace/kohya_ss/./train_db.py", line 481, in <module>
train(args)
File "/workspace/kohya_ss/./train_db.py", line 258, in train
accelerator.init_trackers("dreambooth" if args.log_tracker_name is None else args.log_tracker_name, init_kwargs=init_kwargs)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 564, in _inner
return PartialState().on_main_process(function)(*args, kwargs)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 2096, in init_trackers
self.trackers.append(tracker_init(project_name, init_kwargs.get(str(tracker), {})))
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/tracking.py", line 85, in execute_on_main_process
return function(self, *args, kwargs)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/tracking.py", line 294, in init
self.run = wandb.init(project=self.run_name, kwargs)
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1164, in init
raise e
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1145, in init
run = wi.init()
File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 764, in init
raise error
wandb.errors.AuthenticationError: The API key is either invalid or missing, or the host is incorrect. To resolve this issue, you may try running the 'wandb login --host [hostname]' command. The host defaults to 'https://api.wandb.ai' if not specified. (Error 401: Unauthorized)

Narva•10/10/23, 6:19 PM

I recently tried to start training in Kohya and I got that error, do you know what it could be?

Sandstormleader•10/10/23, 6:19 PM

is there anyway for me to run a seperate instance of automatic1111 so that i can have the old version with working dreambooth as well as a updated version @Dr. Furkan Gözükara

NNarva > : ERROR Error while calling W&B API: user is not logged in (<Response [401]>) ...

Furkan Gözükara SECourses•10/10/23, 6:26 PM

it says wan db api error

1024 sdxl fail again ------------------------------------------------------------ Failures: [1]:

Similar Threads