Software Engineering Courses (SECourses)•15mo ago

I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e-5 (4e-5 was indeed

I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e-5 (4e-5 was indeed too high, but better too high than too low), with 4000 steps, I also activated buckets for multiple aspect ratio's (where each bucket must have at least 7 images). The first 10% of steps are warmup and the LR decreases with a cosine. It's looking like 2e-5 is working the best for my major concept.

VVinch I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e...

ar.1•10/20/24, 11:39 AM

how many images total ?

Aar.1 how many images total ?

VinchOP•10/20/24, 11:41 AM

I'm currently doing stage 1 that has 556 images. The images in stage 1 has high concept and low details, most of these images are AI generated to really get the concept. For stage 2, I have 2551 images, these images have less of the concept but are high in detail and quality, which are mostly production grade photographs and marketing material for movies and such.

VinchOP•10/20/24, 11:41 AM

I don't know what the config will be for stage 2.

VVinch I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e...

ar.1•10/20/24, 11:43 AM

how are you ensuring each bucket has at least 7 ?

VinchOP•10/20/24, 11:43 AM

I made a script. You want?

ar.1•10/20/24, 11:43 AM

yeah sure if you dont mind sharing that would be great

VinchOP•10/20/24, 11:44 AM

The instructions are in the top of the file

fit_in_buckets.py12.22KB

VVinch The instructions are in the top of the file

ar.1•10/20/24, 11:46 AM

amazing thank you bro, ill use it on my next test

VVinch I made a script. You want?

ar.1•10/20/24, 11:47 AM

how did you come up with the idea to have min number of images per bucket ? did you notice it improves anything ?

Aar.1 how did you come up with the idea to have min number of images per bucket ? did ...

VinchOP•10/20/24, 11:55 AM

It is because if you work with batches and you have buckets turned on. A batch will only contain images from a single bucket. So if you have buckets with 1 image. That means that it has to duplicate that single image 7 times to fill a batch, Making it basically ineffective.

VVinch It is because if you work with **batches** and you have __buckets__ turned on. A...

ar.1•10/20/24, 11:55 AM

makes sense

FFurkan Gözükara SECourses way lower than I expect

ar.1•10/20/24, 1:44 PM

what speed where you getting ?

ar.1•10/20/24, 1:44 PM

Ive got H100 going now, its caching latents

Aar.1 what speed where you getting ?

Furkan Gözükara SECourses•10/20/24, 1:55 PM

let me check video part

Furkan Gözükara SECourses•10/20/24, 1:55 PM

i am still editing :d

Furkan Gözükara SECourses•10/20/24, 1:55 PM

21.21 second

Furkan Gözükara SECourses•10/20/24, 1:55 PM

batch size 7

Furkan Gözükara SECourses•10/20/24, 1:56 PM

by the way if dataset cant be divided into 7 exactly, you wont get very accurate speed

FFurkan Gözükara SECourses 21.21 second

ar.1•10/20/24, 2:03 PM

so H100, batch size 12 I getting 16s/it

ar.1•10/20/24, 2:04 PM

65GB GPU memory usage

Aar.1 so H100, batch size 12 I getting 16s/it

Furkan Gözükara SECourses•10/20/24, 2:08 PM

wow nice

Furkan Gözükara SECourses•10/20/24, 2:08 PM

almost 1 second / it per image

ar.1•10/20/24, 2:09 PM

I can probably fit 14 batch size

FFurkan Gözükara SECourses almost 1 second / it per image

ar.1•10/20/24, 2:12 PM

so with H100 at $3 per hour, 10k images with 15 epochs (batch size 12) is gonna cost around $170

Aar.1 so with H100 at $3 per hour, 10k images with 15 epochs (batch size 12) is gonna ...

Furkan Gözükara SECourses•10/20/24, 2:29 PM

looks like way expensive compared to A6000

FFurkan Gözükara SECourses looks like way expensive compared to A6000

ar.1•10/20/24, 2:30 PM

yeah thats what im thinking

Furkan Gözükara SECourses•10/20/24, 2:30 PM

10k images 15 epochs would take 53$

Furkan Gözükara SECourses•10/20/24, 2:30 PM

0.31 cent per hour

ar.1•10/20/24, 2:30 PM

much better

Furkan Gözükara SECourses•10/20/24, 2:31 PM

yep

FFurkan Gözükara SECourses yep

ar.1•10/20/24, 2:32 PM

we cant do multi gpu with A6000 dreambooth finetuning right?

ar.1•10/20/24, 2:33 PM

I wish there was option to add more SSD in massed compute

ar.1•10/20/24, 2:34 PM

256GB on the A6000 isnt that great

Aar.1 we cant do multi gpu with A6000 dreambooth finetuning right?

Furkan Gözükara SECourses•10/20/24, 2:34 PM

yes you need 80 gb gpus

Aar.1 I wish there was option to add more SSD in massed compute

VinchOP•10/20/24, 2:34 PM

I keep having to upload checkpoints and delete them to make room for the next checkpoint

VVinch I keep having to upload checkpoints and delete them to make room for the next ch...

ar.1•10/20/24, 2:34 PM

yeah its a pain haha

ar.1•10/20/24, 2:34 PM

Has anyone tried A100

ar.1•10/20/24, 2:35 PM

I wonder its speed against the H100 as its almost half the price

VVinch I keep having to upload checkpoints and delete them to make room for the next ch...

Furkan Gözükara SECourses•10/20/24, 2:39 PM

you can delete existing apps and models

Furkan Gözükara SECourses•10/20/24, 2:39 PM

shown in new coming video

Furkan Gözükara SECourses•10/20/24, 2:39 PM

also yes i agree 256 GB disk space is very limiting for flux fine tuning :/

FFurkan Gözükara SECourses shown in new coming video

ar.1•10/20/24, 2:40 PM

very eager to watch it, whats the ETA ?

dsienra•10/20/24, 2:40 PM

@Dr. Furkan Gözükara this is cool!!!

Aar.1 very eager to watch it, whats the ETA ?

Furkan Gözükara SECourses•10/20/24, 2:40 PM

still editing

Ddsienra @Dr. Furkan Gözükara this is cool!!!

Furkan Gözükara SECourses•10/20/24, 2:40 PM

this is next hopefully after i published video to test

dsienra•10/20/24, 2:42 PM

sounds very promising!!!

snowmonkey•10/20/24, 2:51 PM

Does anyone have experience creating serverless Comfy API endpoint on RunPod ?
I keep getting "Max retries reached while waiting for image generation" with fresh installation of https://github.com/blib-la/runpod-worker-comfy

dsienra•10/20/24, 4:44 PM

@Dr. Furkan Gözükara Update https://huggingface.co/nyanko7/flux-dev-de-distill Lora training. 3 people same class in 1 lora with regularization, regularization captions just the class, lr 0.00005 unet and Te, very interesting results, high diversity on the class on inference on regular flux-dev, promoting for the trained subjects high resemblance, one curious thing a . on the prompt produces a total different image, for example same seed "Name class" different image than "Name class." is like the . works like a separation in the prompt, more to test, if you will use regularization use only the class for the captions, on previous tests I used captioned regularization and it produces unpredictable and unconcistent results. more things to test

nyanko7/flux-dev-de-distill · Hugging Face

FFurkan Gözükara SECourses this is next hopefully after i published video to test

dsienra•10/20/24, 5:02 PM

Now training De-distilled same dataset with no regularization for testing new output preservation script

Ssnowmonkey Does anyone have experience creating serverless Comfy API endpoint on RunPod ? ...

Furkan Gözükara SECourses•10/20/24, 5:44 PM

this is totally pro stuff. only runpod team may help you for free

I do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e-5 (4e-5 was indeed

Similar Threads

Similar Threads

Similar Threads