Software Engineering Courses (SECourses)•16mo ago

```python 2024-09-03 09:03:32 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0

2024-09-03 09:03:32 INFO     unet dtype: torch.float8_e4m3fn, device: cuda:0                                                                                                        train_network.py:1030
                    INFO     text_encoder dtype: torch.bfloat16, device: cuda:0                                                                                                     train_network.py:1032
                    INFO     text_encoder dtype: torch.bfloat16, device: cpu                                                                                                        train_network.py:1032

epoch 1/50
                    INFO     epoch is incremented. current_epoch: 0, epoch: 1

2024-09-03 09:03:32 INFO     unet dtype: torch.float8_e4m3fn, device: cuda:0                                                                                                        train_network.py:1030
                    INFO     text_encoder dtype: torch.bfloat16, device: cuda:0                                                                                                     train_network.py:1032
                    INFO     text_encoder dtype: torch.bfloat16, device: cpu                                                                                                        train_network.py:1032

epoch 1/50
                    INFO     epoch is incremented. current_epoch: 0, epoch: 1

2024-09-03 09:03:32 INFO     unet dtype: torch.float8_e4m3fn, device: cuda:0                                                                                                        train_network.py:1030
                    INFO     text_encoder dtype: torch.bfloat16, device: cuda:0                                                                                                     train_network.py:1032
                    INFO     text_encoder dtype: torch.bfloat16, device: cpu                                                                                                        train_network.py:1032

epoch 1/50
                    INFO     epoch is incremented. current_epoch: 0, epoch: 1

2024-09-03 09:03:32 INFO     unet dtype: torch.float8_e4m3fn, device: cuda:0                                                                                                        train_network.py:1030
                    INFO     text_encoder dtype: torch.bfloat16, device: cuda:0                                                                                                     train_network.py:1032
                    INFO     text_encoder dtype: torch.bfloat16, device: cpu                                                                                                        train_network.py:1032

epoch 1/50
                    INFO     epoch is incremented. current_epoch: 0, epoch: 1

YobinOP•9/3/24, 8:05 AM

text encoder running on cpu too idk if thats the usual

YobinOP•9/3/24, 8:06 AM

never ran a diffusion transformer before, apologies for the dumb questions lol im a sdxl guy

YYobin you think 200 epoch is necessary for 14 images only of one anime character? im d...

Furkan Gözükara SECourses•9/3/24, 8:09 AM

you have 52k steps

Furkan Gözükara SECourses•9/3/24, 8:09 AM

best to rent 4x gpu and use 4x config :d

YYobin ```python 2024-09-03 09:03:32 INFO unet dtype: torch.float8_e4m3fn, device: ...

Furkan Gözükara SECourses•9/3/24, 8:10 AM

this is normal

Furkan Gözükara SECourses•9/3/24, 8:10 AM

we dont train T5 text encoder

FFurkan Gözükara SECourses best to rent 4x gpu and use 4x config :d

YobinOP•9/3/24, 8:12 AM

i can do x4 in runpod, do i need to setup anything aside from the config??

YYobin i can do x4 in runpod, do i need to setup anything aside from the config??

Furkan Gözükara SECourses•9/3/24, 8:12 AM

nope

Furkan Gözükara SECourses•9/3/24, 8:12 AM

i made video but still editing it :d

Furkan Gözükara SECourses•9/3/24, 8:13 AM

dont forget to divide epoch count to 4

Furkan Gözükara SECourses•9/3/24, 8:13 AM

because in 1 step it will do 4 steps

YobinOP•9/3/24, 8:13 AM

aight ill do one gpu first under rank 3 then wait for ur vid

  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
steps:   0%|▍                                                                                                   | 28/6500 [09:25<36:18:15, 20.19s/it, avr_loss=0.207]

  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
steps:   0%|▍                                                                                                   | 28/6500 [09:25<36:18:15, 20.19s/it, avr_loss=0.207]

YobinOP•9/3/24, 8:13 AM

36 hours not bad

YobinOP•9/3/24, 8:13 AM

2 batch size - x1 A40 gpu

YobinOP•9/3/24, 8:16 AM

not sure for flux but what about increasing "Gradient accumulate steps" ? probably at least - 2, you think would help?

YYobin not sure for flux but what about increasing "Gradient accumulate steps" ? probab...

Furkan Gözükara SECourses•9/3/24, 8:19 AM

no i dont suggest

Furkan Gözükara SECourses•9/3/24, 8:19 AM

batch size bigger = lower quality

Furkan Gözükara SECourses•9/3/24, 8:19 AM

best is batch size 2 max

Furkan Gözükara SECourses•9/3/24, 8:19 AM

and multi gpu

Furkan Gözükara SECourses•9/3/24, 8:19 AM

also you need to change LR

YYobin aight ill do one gpu first under rank 3 then wait for ur vid```python with tor...

Furkan Gözükara SECourses•9/3/24, 8:19 AM

this is so slow it should get better

Furkan Gözükara SECourses•9/3/24, 8:19 AM

on RTX A6000 on massed compute fast config of rank 1 gets like 4.5 second it - 5 second it on multi gpu

YobinOP•9/3/24, 8:26 AM

aight let me check

YobinOP•9/3/24, 8:27 AM

imma make the LR more aggressive

YobinOP•9/3/24, 8:27 AM

0.00005 atm

YYobin imma make the LR more aggressive

Furkan Gözükara SECourses•9/3/24, 8:28 AM

ye with multi gpu and batch size you need to increase

Furkan Gözükara SECourses•9/3/24, 8:28 AM

i explained logic in coming video :d

FFurkan Gözükara SECourses i explained logic in coming video :d

YobinOP•9/3/24, 8:32 AM

ok thank u!

FFurkan Gözükara SECourses ye with multi gpu and batch size you need to increase

GomezBro•9/3/24, 8:53 AM

brother whats the proper structure to do 1000 images with an accompanying description in text file form. I think I want to do another lora of 1000 images with descriptions this time to see if it trains better and to see if those trained words allow me to generate the things in the descriptions

GomezBro•9/3/24, 8:53 AM

I dont need help making the description, I will make a python script for that since I already have metadata for each file

GomezBro•9/3/24, 8:54 AM

just want to know the proper structure and how to make sure Kohya knows the description is there or is it automatic if the txt file has the same name as the image?

GGomezBro just want to know the proper structure and how to make sure Kohya knows the desc...

Furkan Gözükara SECourses•9/3/24, 9:00 AM

filename.jpg + filename.txt

Furkan Gözükara SECourses•9/3/24, 9:00 AM

in the same folder

Furkan Gözükara SECourses•9/3/24, 9:00 AM

select captions extension txt

Furkan Gözükara SECourses•9/3/24, 9:00 AM

thats it

Furkan Gözükara SECourses•9/3/24, 9:00 AM

it doesnt have to be jpg can be png too

GomezBro•9/3/24, 9:00 AM

thats it? where is that captions extention txt?

GomezBro•9/3/24, 9:00 AM

png is what I do

GomezBro•9/3/24, 9:00 AM

there is a switch or its automatic?

GGomezBro thats it? where is that captions extention txt?

Furkan Gözükara SECourses•9/3/24, 9:01 AM

that is it

Furkan Gözükara SECourses•9/3/24, 9:01 AM

my configs are by default .txt

Furkan Gözükara SECourses•9/3/24, 9:01 AM

do ctrl f and type .txt to see

GomezBro•9/3/24, 9:02 AM

Furkan Gözükara SECourses•9/3/24, 9:02 AM

example style dataset published

Furkan Gözükara SECourses•9/3/24, 9:02 AM

https://huggingface.co/MonsterMMORPG/3D-Cartoon-Style-FLUX/tree/main/Training-Dataset

MonsterMMORPG/3D-Cartoon-Style-FLUX at main

Furkan Gözükara SECourses•9/3/24, 9:02 AM

true

GomezBro•9/3/24, 9:04 AM

so how does it do it with no captions now? I do see when I do dataset preparations it does add 2 additional files per image however, in the image folder. so as long as the description matches the filename we are good. also do i put the instance prompt and class prompt at the beginning of each description. I will set it up now

GomezBro•9/3/24, 9:04 AM

thanks

GGomezBro so how does it do it with no captions now? I do see when I do dataset preparatio...

Furkan Gözükara SECourses•9/3/24, 9:05 AM

reads folder name

Furkan Gözükara SECourses•9/3/24, 9:05 AM

you really should watch video

```python 2024-09-03 09:03:32 INFO unet dtype: torch.float8_e4m3fn, device: cuda:0

Similar Threads