like is the speed improvement worth it you think
like is the speed improvement worth it you think
Fwell
Aits almost 10x the cost
Fi think L40S and batch size 7 most sense making
Fbut on runpod speed was
Fway lower than I expect
Fmaybe it was that pod
Fi didnt test h100 speed
Fyou should test effective step speed
Aill test it later today and report back
VI do fine-tuning with dreambooth batch-7, I'm currently testing with 2e-5 and 1e-5 (4e-5 was indeed too high, but better too high than too low), with 4000 steps, I also activated buckets for multiple aspect ratio's (where each bucket must have at least 7 images). The first 10% of steps are warmup and the LR decreases with a cosine. It's looking like 2e-5 is working the best for my major concept.
Ahow many images total ?
VI'm currently doing stage 1 that has 556 images. The images in stage 1 has high concept and low details, most of these images are AI generated to really get the concept. For stage 2, I have 2551 images, these images have less of the concept but are high in detail and quality, which are mostly production grade photographs and marketing material for movies and such.
VI don't know what the config will be for stage 2.
Ahow are you ensuring each bucket has at least 7 ?
VI made a script. You want?
Ayeah sure if you dont mind sharing that would be great
VThe instructions are in the top of the file
fit_in_buckets.py12.22KB
Aamazing thank you bro, ill use it on my next test
Ahow did you come up with the idea to have min number of images per bucket ? did you notice it improves anything ?
VIt is because if you work with batches and you have buckets turned on. A batch will only contain images from a single bucket. So if you have buckets with 1 image. That means that it has to duplicate that single image 7 times to fill a batch, Making it basically ineffective.
Amakes sense
Awhat speed where you getting ?
AIve got H100 going now, its caching latents
Flet me check video part
Fi am still editing :d
F21.21 second
Fbatch size 7
Fby the way if dataset cant be divided into 7 exactly, you wont get very accurate speed
Aso H100, batch size 12 I getting 16s/it
A65GB GPU memory usage
Fwow nice
Falmost 1 second / it per image
AI can probably fit 14 batch size
Aso with H100 at $3 per hour, 10k images with 15 epochs (batch size 12) is gonna cost around $170
Flooks like way expensive compared to A6000
Ayeah thats what im thinking
F10k images 15 epochs would take 53$
F0.31 cent per hour
Amuch better
Fyep
Awe cant do multi gpu with A6000 dreambooth finetuning right?
AI wish there was option to add more SSD in massed compute
A256GB on the A6000 isnt that great
Fyes you need 80 gb gpus
VI keep having to upload checkpoints and delete them to make room for the next checkpoint
Ayeah its a pain haha
AHas anyone tried A100
AI wonder its speed against the H100 as its almost half the price
Fyou can delete existing apps and models
Fshown in new coming video
