Software Engineering Courses (SECourses)•16mo ago

even with T5 training captions yields lower quality

FFurkan Gözükara SECourses Click to see attachment

AbuYunus•9/9/24, 10:15 AM

best way to travel

AAbuYunus best way to travel 🙂

Furkan Gözükara SECoursesOP•9/9/24, 10:16 AM

yep

FFurkan Gözükara SECourses even with T5 training captions yields lower quality

magical•9/9/24, 10:27 AM

You are getting better results when you train without captions?

Mmagical You are getting better results when you train without captions?

Furkan Gözükara SECoursesOP•9/9/24, 10:31 AM

yes

Furkan Gözükara SECoursesOP•9/9/24, 10:31 AM

i will post on patreon and reddit

FFurkan Gözükara SECourses i will post on patreon and reddit

magical•9/9/24, 10:32 AM

one more thing there is no point in checking Apply T5 Attention Mask when you are training without captions right?

Mmagical one more thing there is no point in checking Apply T5 Attention Mask when you ar...

Furkan Gözükara SECoursesOP•9/9/24, 10:38 AM

i think iy very slightly improves

Furkan Gözükara SECoursesOP•9/9/24, 10:38 AM

also with T5 training i think unet can go lower

Furkan Gözükara SECoursesOP•9/9/24, 10:38 AM

it brings some quality

Furkan Gözükara SECoursesOP•9/9/24, 10:38 AM

but in some images resemblance is lost

Furkan Gözükara SECoursesOP•9/9/24, 10:39 AM

so it is a choice

Furkan Gözükara SECoursesOP•9/9/24, 10:39 AM

i am analyinz

Maxivi•9/9/24, 12:39 PM

@Dr. Furkan Gözükara T5 is mainly for text trainings, t5 not worth training at all

Maxivi•9/9/24, 12:40 PM

i did about 60+ trainings with it

Maxivi•9/9/24, 12:40 PM

it'll work amazing if you're training text

MMaxivi @Dr. Furkan Gözükara T5 is mainly for text trainings, t5 not worth training at a...

Furkan Gözükara SECoursesOP•9/9/24, 12:45 PM

i see

Furkan Gözükara SECoursesOP•9/9/24, 12:45 PM

what do you train text

Furkan Gözükara SECoursesOP•9/9/24, 12:46 PM

with cpu offloading it uses 23.34 gb

Furkan Gözükara SECoursesOP•9/9/24, 12:46 PM

24 gb may train

Furkan Gözükara SECoursesOP•9/9/24, 12:46 PM

and 22.44 gb

Furkan Gözükara SECoursesOP•9/9/24, 12:46 PM

secondary config

FFurkan Gözükara SECourses what do you train text

Maxivi•9/9/24, 12:46 PM

kohya mentioned to me in dms.

Maxivi•9/9/24, 12:46 PM

hes added different learning rates for t5 and clip l

Maxivi•9/9/24, 12:47 PM

so he said you'll need to lower t5
to 1e-7 or 1e-6

MMaxivi kohya mentioned to me in dms.

Furkan Gözükara SECoursesOP•9/9/24, 12:49 PM

actually even 05-05 even didnt cook the model at all

Furkan Gözükara SECoursesOP•9/9/24, 12:49 PM

i mean the T5

Furkan Gözükara SECoursesOP•9/9/24, 12:50 PM

although i didnt do text training

Furkan Gözükara SECoursesOP•9/9/24, 12:50 PM

i compared ohwx man vs full captioned

Maxivi•9/9/24, 12:50 PM

t5 wont do much for people training

Maxivi•9/9/24, 12:51 PM

t5 is a text to text model.

you may get better results adjusting the clip l

kohya suggested to me having clip l 1/10

so for your config unet 5e-05

clip l 5e-06

Furkan Gözükara SECoursesOP•9/9/24, 12:52 PM

i tried every clip LR already :d

Maxivi•9/9/24, 12:59 PM

but then again you're only training with ohwx man so it wont do much

MMaxivi but then again you're only training with ohwx man so it wont do much

Furkan Gözükara SECoursesOP•9/9/24, 12:59 PM

i tried full joy captions too

Furkan Gözükara SECoursesOP•9/9/24, 12:59 PM

300 tokens

FFurkan Gözükara SECourses i tried full joy captions too

Maxivi•9/9/24, 1:00 PM

not worth it, if you're training that many tokens you need to lower all your rates and train for a long time

Furkan Gözükara SECoursesOP•9/9/24, 1:00 PM

but may depends on dataset

Furkan Gözükara SECoursesOP•9/9/24, 1:00 PM

i will focus on fine tuning

FFurkan Gözükara SECourses i will focus on fine tuning 😄

Maxivi•9/9/24, 1:01 PM

tried on a100 got quite good results, idk if he fixed the extract lora thing yet, so you need to use higher weight when using it

MMaxivi it'll work amazing if you're training text

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/9/24, 1:05 PM

How about training clothes with text on them?

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 How about training clothes with text on them?

Maxivi•9/9/24, 1:17 PM

That could work. it'll work good for people who are holding up signs, text on clothings, product placements

Maxivi•9/9/24, 1:17 PM

but for people training t5 is not worth it at all

MMaxivi That could work. it'll work good for people who are holding up signs, text on cl...

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/9/24, 1:18 PM

to train using t5 do I just use doc's config and select train t5 in the GUI?

�𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮 to train using t5 do I just use doc's config and select train t5 in the GUI?

Maxivi•9/9/24, 1:18 PM

idk if he updated his config yet

Maxivi•9/9/24, 1:18 PM

but you have to enable it in the advanced parameters

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/9/24, 1:19 PM

Okay

Furkan Gözükara SECoursesOP•9/9/24, 1:20 PM

i will update configs hopefully soon

MMaxivi but you have to enable it in the advanced parameters

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/9/24, 1:20 PM

this right?

Maxivi•9/9/24, 1:21 PM

correct and enter this into your additonal parameters, cause i think the cpu offload checkpointing is not working.

--cpu_offlload_checkpointing

MMaxivi correct and enter this into your additonal parameters, cause i think the cpu off...

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/9/24, 1:22 PM

Thanks

MMaxivi correct and enter this into your additonal parameters, cause i think the cpu off...

�

𝗡𝗶𝘁𝗶𝗻 𝗣𝗮𝗿𝗲𝘃𝗮•9/9/24, 1:23 PM

Can I DM you regarding one more thing?

even with T5 training captions yields lower quality

Similar Threads