Software Engineering Courses (SECourses)•15mo ago

maybe needs more

ar.1OP•10/20/24, 11:30 AM

with very low lr

ar.1OP•10/20/24, 11:30 AM

like 100 epoch

Aar.1 maybe needs more

Furkan Gözükara SECourses•10/20/24, 11:30 AM

yep

Furkan Gözükara SECourses•10/20/24, 11:30 AM

possibly

FFurkan Gözükara SECourses yep

ar.1OP•10/20/24, 11:30 AM

what was the reason you stopped at 5 ?

Aar.1 what was the reason you stopped at 5 ?

Furkan Gözükara SECourses•10/20/24, 11:31 AM

i had to stop to do trainings for fine tuning tutorial

FFurkan Gözükara SECourses i had to stop to do trainings for fine tuning tutorial 😄

ar.1OP•10/20/24, 11:32 AM

ah haha

ar.1OP•10/20/24, 11:32 AM

is there much benefit to using H100 vs A6000 for this

ar.1OP•10/20/24, 11:32 AM

like is the speed improvement worth it you think

Aar.1 like is the speed improvement worth it you think

Furkan Gözükara SECourses•10/20/24, 11:32 AM

well

ar.1OP•10/20/24, 11:33 AM

its almost 10x the cost

Furkan Gözükara SECourses•10/20/24, 11:33 AM

i think L40S and batch size 7 most sense making

Furkan Gözükara SECourses•10/20/24, 11:33 AM

but on runpod speed was

Furkan Gözükara SECourses•10/20/24, 11:33 AM

way lower than I expect

Furkan Gözükara SECourses•10/20/24, 11:33 AM

maybe it was that pod

Furkan Gözükara SECourses•10/20/24, 11:34 AM

i didnt test h100 speed

Furkan Gözükara SECourses•10/20/24, 11:34 AM

you should test effective step speed

FFurkan Gözükara SECourses i didnt test h100 speed

ar.1OP•10/20/24, 11:34 AM

ill test it later today and report back

Aar.1 ye I just testing with something for now, but my biggest question is how many st...

Vinch•10/20/24, 11:35 AM

I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e-5 (4e-5 was indeed too high, but better too high than too low), with 4000 steps, I also activated buckets for multiple aspect ratio's (where each bucket must have at least 7 images). The first 10% of steps are warmup and the LR decreases with a cosine. It's looking like 2e-5 is working the best for my major concept.

VVinch I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e...

ar.1OP•10/20/24, 11:39 AM

how many images total ?

Aar.1 how many images total ?

Vinch•10/20/24, 11:41 AM

I'm currently doing stage 1 that has 556 images. The images in stage 1 has high concept and low details, most of these images are AI generated to really get the concept. For stage 2, I have 2551 images, these images have less of the concept but are high in detail and quality, which are mostly production grade photographs and marketing material for movies and such.

Vinch•10/20/24, 11:41 AM

I don't know what the config will be for stage 2.

VVinch I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e...

ar.1OP•10/20/24, 11:43 AM

how are you ensuring each bucket has at least 7 ?

Vinch•10/20/24, 11:43 AM

I made a script. You want?

ar.1OP•10/20/24, 11:43 AM

yeah sure if you dont mind sharing that would be great

Vinch•10/20/24, 11:44 AM

The instructions are in the top of the file

fit_in_buckets.py12.22KB

VVinch The instructions are in the top of the file

ar.1OP•10/20/24, 11:46 AM

amazing thank you bro, ill use it on my next test

VVinch I made a script. You want?

ar.1OP•10/20/24, 11:47 AM

how did you come up with the idea to have min number of images per bucket ? did you notice it improves anything ?

Aar.1 how did you come up with the idea to have min number of images per bucket ? did ...

Vinch•10/20/24, 11:55 AM

It is because if you work with batches and you have buckets turned on. A batch will only contain images from a single bucket. So if you have buckets with 1 image. That means that it has to duplicate that single image 7 times to fill a batch, Making it basically ineffective.

VVinch It is because if you work with **batches** and you have __buckets__ turned on. A...