OK I shutoff the L40S, luckily there is a 4GPU A6000 now

G

GomezBroOP•9/4/24, 10:24 PM

i took it and will test

GGomezBro i took it and will test

F

Furkan Gözükara SECourses•9/4/24, 10:24 PM

sure

F

Furkan Gözükara SECourses•9/4/24, 10:24 PM

good idea

F

Furkan Gözükara SECourses•9/4/24, 10:24 PM

once you make it work

F

Furkan Gözükara SECourses•9/4/24, 10:24 PM

you can move to l40s

GGomezBro OK I shutoff the L40S, luckily there is a 4GPU A6000 now

F

Furkan Gözükara SECourses•9/4/24, 10:25 PM

you can even move cached files

F

Furkan Gözükara SECourses•9/4/24, 10:25 PM

it shouldnt recache on l40s

G

GomezBroOP•9/4/24, 10:26 PM

OK I will copy the Lora directory over onced cached

FFurkan Gözükara SECourses what gpu which config how much vram used before starting training

N

Neuromasters•9/4/24, 10:41 PM

3 rank
4090 gpu loсal

NNeuromasters 3 rank 4090 gpu loсal

F

Furkan Gözükara SECourses•9/4/24, 10:42 PM

it shoudl work

F

Furkan Gözükara SECourses•9/4/24, 10:42 PM

working or not?

N

Neuromasters•9/4/24, 10:43 PM

operate

NNeuromasters operate

F

Furkan Gözükara SECourses•9/4/24, 10:43 PM

nice

F

Furkan Gözükara SECourses•9/4/24, 10:43 PM

working good

F

Furkan Gözükara SECourses•9/4/24, 10:43 PM

if you use rank 4 will be faster

F

Furkan Gözükara SECourses•9/4/24, 10:43 PM

i am testing torch 2.4.1 right now

F

Furkan Gözükara SECourses•9/4/24, 10:43 PM

it may boost even more

F

Furkan Gözükara SECourses•9/4/24, 10:49 PM

looks like 2.4.1 still doesnt fix :/

F

Furkan Gözükara SECourses•9/4/24, 10:59 PM

ok torch 2.5 still best

F

Furkan Gözükara SECourses•9/4/24, 10:59 PM

for windows

F

Furkan Gözükara SECourses•9/4/24, 10:59 PM

improves both rank 3,4,5,6 and all

F

Furkan Gözükara SECourses•9/4/24, 11:00 PM

i am gonna test something else now a new stuff

F

Furkan Gözükara SECourses•9/4/24, 11:10 PM

rtx 3090 and rtx 3060 faster ranks

F

Furkan Gözükara SECourses•9/4/24, 11:10 PM

min speeds on windows

W

Websteria•9/4/24, 11:36 PM

Is there a way to change it to use 512x512 images as the source?

FFurkan Gözükara SECourses you can even move cached files

G

GomezBroOP•9/4/24, 11:39 PM

still doesnt work, 1 GPU dies in same spot

G

GomezBroOP•9/4/24, 11:39 PM

I wasted 20$ on multiple instances, I will wait until you test 10,000 IMAGES with Captions and figure out the correct settings

G

GomezBroOP•9/4/24, 11:40 PM

it does not work

M

Maxivi•9/4/24, 11:54 PM

@Dr. Furkan Gözükara t5 training is out

M

Maxivi•9/4/24, 11:57 PM

getting way better results now with captions

M

Maxivi•9/4/24, 11:57 PM

https://tenor.com/view/kermit-the-frog-meme-memes-gif-24277198

Tenor

WWebsteria Is there a way to change it to use 512x512 images as the source?

F

Furkan Gözükara SECourses•9/5/24, 12:03 AM

yep

F

Furkan Gözükara SECourses•9/5/24, 12:03 AM

just change 1024 to 512 on gradio

GGomezBro still doesnt work, 1 GPU dies in same spot

F

Furkan Gözükara SECourses•9/5/24, 12:03 AM

you should post it as error on kohya meanwhile

F

Furkan Gözükara SECourses•9/5/24, 12:03 AM

maybe it is related to him

MMaxivi @Dr. Furkan Gözükara t5 training is out

F

Furkan Gözükara SECourses•9/5/24, 12:03 AM

ye i saw

F

Furkan Gözükara SECourses•9/5/24, 12:03 AM

bmaltais also merged nice

F

Furkan Gözükara SECourses•9/5/24, 12:04 AM

currently uses same LR i presume

M

Maxivi•9/5/24, 12:06 AM

you can't cache text encoder on bmaltais

MMaxivi you can't cache text encoder on bmaltais

F

Furkan Gözükara SECourses•9/5/24, 12:06 AM

what do you mean

M

Maxivi•9/5/24, 12:06 AM

File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 514, in <module> trainer.train(args) File "D:\kohya_ss\sd-scripts\train_network.py", line 441, in train self.post_process_network(args, accelerator, network, text_encoders, unet) File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 168, in post_process_network raise ValueError( ValueError: T5XXL is trained, so cache_text_encoder_outputs cannot be used / T5XXL学習時はcache_text_encoder_outputsは使用できません

M

Maxivi•9/5/24, 12:07 AM

i think bmaltais doesn't work correctly, because on sd scripts you can train 1024 with t5 on 22-23gb vram

i think bmaltais doesn't work correctly, because on sd scripts you can train 1024 with t5 on 22-23gb vram

i think bmaltais doesn't work correctly, because on sd scripts you can train 1024 with t5 on 22-23gb vram

i think bmaltais doesn't work correctly, because on sd scripts you can train 1024 with t5 on 22-23gb vram

MMaxivi File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 514, in <module> ...

F

Furkan Gözükara SECourses•9/5/24, 12:08 AM

i think cache is for when we dont train t5

F

Furkan Gözükara SECourses•9/5/24, 12:08 AM

not sure

M

Maxivi•9/5/24, 12:08 AM

yeah without it, you need 48gb vram gpu

M

Maxivi•9/5/24, 12:12 AM

got it working with split mode single

FFurkan Gözükara SECourses yep

W

Websteria•9/5/24, 1:10 AM

Where do I change the 1024 to 512?

WWebsteria Where do I change the 1024 to 512?

F

Furkan Gözükara SECourses•9/5/24, 1:12 AM

search this keyword on gradio of kohya : Max resolution

FFurkan Gözükara SECourses search this keyword on gradio of kohya : Max resolution

W

Websteria•9/5/24, 1:13 AM

I did that, but it didn't do any training then.

W

Websteria•9/5/24, 1:15 AM

INFO [Dataset 0] train_util.py:2326
INFO caching latents with caching strategy. train_util.py:984
INFO checking cache validity... train_util.py:994
100%|██████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<?, ?it/s]
INFO caching latents... train_util.py:1038
0%| | 0/14 [00:00<?, ?it/s]18:14:23-407701 INFO Training has ended.

Similar Threads