Software Engineering Courses (SECourses)•17mo ago

Why are my lora files so big (2Gb)? I trained them with Network Rank = 128, what value should I set

Why are my lora files so big (2Gb)?
I trained them with Network Rank = 128, what value should I set to get ~200mb files?

DragoyOP•8/30/24, 11:23 PM

used the Rank_3_18246MB_Slow configuration

Ddsienra I'm using my actual name "Diego" I didn't try with a unique token that are not ...

Furkan Gözükara SECourses•8/30/24, 11:27 PM

well the thing is we use something rarely exists so that it can overrwite data

Furkan Gözükara SECourses•8/30/24, 11:27 PM

but t5 thing is new in the workflow opposed to sdxl

DDragoy Why are my lora files so big (2Gb)? I trained them with Network Rank = 128, what...

Furkan Gözükara SECourses•8/30/24, 11:28 PM

because we save as float

Furkan Gözükara SECourses•8/30/24, 11:28 PM

save as fp16

Furkan Gözükara SECourses•8/30/24, 11:28 PM

also we save rank 128 :d

FFurkan Gözükara SECourses well the thing is we use something rarely exists so that it can overrwite data

dsienra•8/30/24, 11:30 PM

I know but now with T5 in the equation that is frozen, how reacts to tokens that are not present on it?

dsienra•8/30/24, 11:30 PM

that's the question???

dsienra•8/30/24, 11:31 PM

really don't know

dsienra•8/30/24, 11:31 PM

there is a way to know if a token is on T5 or not?

Ddsienra I know but now with T5 in the equation that is frozen, how reacts to tokens that...

Furkan Gözükara SECourses•8/30/24, 11:51 PM

everything get tokenized

Furkan Gözükara SECourses•8/30/24, 11:51 PM

so it exists

Furkan Gözükara SECourses•8/30/24, 11:51 PM

but it is single token or not unknown

Furkan Gözükara SECourses•8/30/24, 11:51 PM

i asked this to swarmui dev and he said we dont have t5 tokenizer :d

Furkan Gözükara SECourses•8/30/24, 11:51 PM

like clip

Furkan Gözükara SECourses•8/30/24, 11:51 PM

time to work hard :/

Furkan Gözükara SECourses•8/30/24, 11:51 PM

FFurkan Gözükara SECourses c:/kohya_ai

GomezBro•8/30/24, 11:59 PM

is it c:/kohya_ai/kohya_ss correct? batch files in main c:/kohya_ai correct?

Uuisato Great! Changed names and installed said the DLL and it works. Thanks guys.

GomezBro•8/31/24, 12:01 AM

is it c:/kohya_ai/kohya_ss correct? batch files in main c:/kohya_ai correct? and then run the files creating the /kohya_ss folder right? I have this issue too, on 2 machines

GGomezBro is it c:/kohya_ai/kohya_ss correct? batch files in main c:/kohya_ai correct?

Furkan Gözükara SECourses•8/31/24, 12:02 AM

yes

GGomezBro is it c:/kohya_ai/kohya_ss correct? batch files in main c:/kohya_ai correct? a...

Furkan Gözükara SECourses•8/31/24, 12:02 AM

this is accurate name should work

GomezBro•8/31/24, 12:02 AM

Ok will try rn its Windows 10 PRO too not 11

GomezBro•8/31/24, 12:02 AM

ok will update shortly

GGomezBro ok will update shortly

Furkan Gözükara SECourses•8/31/24, 12:03 AM

sure let us know

hunterwolf602•8/31/24, 12:09 AM

Good evening everyone! @Dr. Furkan Gözükara, will there be a Flux training video for RunPod?

Hhunterwolf602 Good evening everyone! @Dr. Furkan Gözükara, will there be a Flux training video...

Furkan Gözükara SECourses•8/31/24, 12:10 AM

yep

Furkan Gözükara SECourses•8/31/24, 12:10 AM

editing right now

Furkan Gözükara SECourses•8/31/24, 12:10 AM

FFurkan Gözükara SECourses editing right now

hunterwolf602•8/31/24, 12:54 AM

Thank you, Dr.!

Furkan Gözükara SECourses•8/31/24, 1:55 AM

Training a FLUX LoRA for a client. 24 images. on 4x RTX A6000 GPU. Total 240 epochs. Effective batch size 4 and step count is 1440

Furkan Gözükara SECourses•8/31/24, 1:55 AM

i also found a new info regarding multi gpu training will update configs

Furkan Gözükara SECourses•8/31/24, 2:05 AM

i am moving to the city tomorrow hopefully. time to get 1 gbit internet

Furkan Gözükara SECourses•8/31/24, 2:07 AM

time to start 8 more test on flux lora LR

Furkan Gözükara SECourses•8/31/24, 2:39 AM

testing more trainings of text encoder - clip L

FFurkan Gözükara SECourses Training a FLUX LoRA for a client. 24 images. on 4x RTX A6000 GPU. Total 240 epo...

GomezBro•8/31/24, 2:39 AM

How long did it take? Size1024x1024? I have the A6000. And what is that EFFECTIVE BATCH SIZE 4 Setting at?

GGomezBro How long did it take? Size1024x1024? I have the A6000. And what is that EFFE...

Furkan Gözükara SECourses•8/31/24, 2:39 AM

it is almost linear increase

Furkan Gözükara SECourses•8/31/24, 2:39 AM

with slow work flow

Furkan Gözükara SECourses•8/31/24, 2:39 AM

9.41 second it

GomezBro•8/31/24, 2:40 AM

image res size? and the batch setting?

GomezBro•8/31/24, 2:40 AM

i did torch 2.5 update

Furkan Gözükara SECourses•8/31/24, 2:40 AM

so lets say 3000 steps takes 9.41 * 3000 / 4 / 60 = 117 minutes

Furkan Gözükara SECourses•8/31/24, 2:40 AM

it is 1024x1024

GomezBro•8/31/24, 2:40 AM

and i had to add the dll in system32 to fix my fbgemm.dll errors

GGomezBro and i had to add the dll in system32 to fix my fbgemm.dll errors

Furkan Gözükara SECourses•8/31/24, 2:40 AM

ye it is a solution

Furkan Gözükara SECourses•8/31/24, 2:41 AM

if you go with fast settings it takes 60 minutes for 3000 steps

Furkan Gözükara SECourses•8/31/24, 2:41 AM

steps count becoming 750 since 4x gpu

Furkan Gözükara SECourses•8/31/24, 2:41 AM

and effective batch size being 4

GomezBro•8/31/24, 2:41 AM

where do you set steps? that is controlled only by the amount of Images you have right?

GomezBro•8/31/24, 2:41 AM

I only have 1 GPU though

GGomezBro where do you set steps? that is controlled only by the amount of Images you hav...

Furkan Gözükara SECourses•8/31/24, 2:42 AM

for 1 gpu you dont need to change anything

Why are my lora files so big (2Gb)? I trained them with Network Rank = 128, what value should I set

Similar Threads