each training is like 7 hours
each training is like 7 hours
V
Vvery long mine are usually less than 1 hour
F
Fi am updating versions on the server now
FTo not wait an entire day I am going to train on 2x RTX 4080 , 1024x1024 and 512x512 resolutions
Results will be in 8 - 9 hours hopefully published - got to sleep
Results will be in 8 - 9 hours hopefully published - got to sleep
Fdmn huge vram drop and speed increase with newest driver
F
Ffrom 44 gb to 27 staggering
Ffrom 8.4 second to 4.4 second
Ftesting 24 gb config
Fhuge news : https://www.reddit.com/r/StableDiffusion/comments/1ey7nwy/flux_lora_rank_128_kohya_ss_gui_training_for_8bit/
Reddit
Explore this post and more from the StableDiffusion community
Aupvoted
Fthanks. testing right now
F512px uses only 2gb lesser vram
Fbut 2x faster
KDid you update driver manually on massed compute?
Fyep
Falso i had to prevent kohya to overwrite it
Fmessage.txt4.22KB
Fgui.sh file
RHi @Dr. Furkan Gözükara, if today is your birthday and you have unlimited budget, which GPU would you buy?
Ffrom consumer grade rtx 4090
GGuys I am training on my 3090 ti 24GB,(I also own a 48GB A6000 should I use that or its the same result?) Do I have to change anything on instance prompts or parameters? Those are all blank.
All i do is run kohya, load the 24gb json settings, change directories to my tensors, images and then hit train, i get errors:
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\queueing.py", line 532, in process_events response = await route_utils.call_process_api(
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api
result = await self.call_function(
\Kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function
prediction = await anyio.to_thread.run_sync(
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\anyio_backends_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\anyio_backends_asyncio.py", line 859, in run
result = context.run(func, args)
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\utils.py", line 832, in wrapper
response = f(args, **kwargs)
\Desktop\Kohya\kohya_ss\kohya_gui\lora_gui.py", line 1171, in train_model
"network_module": network_module,
\Desktop\Kohya\kohya_ss\kohya_gui\lora_gui.py", line 1171, in train_model
"network_module": network_module,
UnboundLocalError: local variable 'network_module' referenced before assignment
All i do is run kohya, load the 24gb json settings, change directories to my tensors, images and then hit train, i get errors:
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\queueing.py", line 532, in process_events response = await route_utils.call_process_api(
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api
result = await self.call_function(
\Kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function
prediction = await anyio.to_thread.run_sync(
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\anyio_backends_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\anyio_backends_asyncio.py", line 859, in run
result = context.run(func, args)
\Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\utils.py", line 832, in wrapper
response = f(args, **kwargs)
\Desktop\Kohya\kohya_ss\kohya_gui\lora_gui.py", line 1171, in train_model
"network_module": network_module,
\Desktop\Kohya\kohya_ss\kohya_gui\lora_gui.py", line 1171, in train_model
"network_module": network_module,
UnboundLocalError: local variable 'network_module' referenced before assignment
GAny other settings I need to change before I hit train button?
Pare you using KohyaSS or SD scripts without GUI ? i cant find Checkbox for Flux in Kohya SD3-FLux branch. Only XL and SD 3 ones.
Mloaad the flux model fist, and make sure you're in the sd3 flux branch
V@Dr. Furkan Gözükara i forgot I could extract lora from ckpt 
V@Dr. Furkan Gözükara Traceback (most recent call last):
File "/opt/conda/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/kaggle/working/kohya_ss/sd-scripts/sdxl_train.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-08-22_10:17:12
host : 979028f82d51
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1222)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
10:17:13-143688 INFO Training has ended. , what is wrong, i use P100
File "/opt/conda/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/kaggle/working/kohya_ss/sd-scripts/sdxl_train.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-08-22_10:17:12
host : 979028f82d51
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1222)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
10:17:13-143688 INFO Training has ended. , what is wrong, i use P100
Vanother question i am training sdxl lora, directly, but why the progress is like this
Fdid you switch to flux branch and loaded as lora?
Fi use with gui
Fnot sure
Ftesting new flux trainings
GI dont even see that in the instructions, that guide on Patreon doesnt say any of this
Fit says you probably missed
F,
Fyou should read that post very very carefully
GSo after that, put proper directories then hit train? no changes on parameters etc?
Fyes currently it will train up to 200 epoch
Fyou can chance save checkpoint frequency
Gi dont set steps nothing?
Falso i suggest manual venv torch version update explained in top
Fwe dont have steps since not using reg images
GOK ill check, and just straight images right dont need to describe them none of that
G20 images or 30 or 40? whats best bro
Gand some are far/closer differnent angles of the subject
F512px lower quality as expected
Feven 10gb config working perfect amazing
Fdont know that yet
