Software Engineering Courses (SECourses)•15mo ago

It is because if you work with batches and you have buckets turned on. A batch will only

It is because if you work with batches and you have buckets turned on. A batch will only contain images from a single bucket. So if you have buckets with 1 image. That means that it has to duplicate that single image 7 times to fill a batch, Making it basically ineffective.

VVinch It is because if you work with **batches** and you have __buckets__ turned on. A...

ar.1•10/20/24, 11:55 AM

makes sense

FFurkan Gözükara SECourses way lower than I expect

ar.1•10/20/24, 1:44 PM

what speed where you getting ?

ar.1•10/20/24, 1:44 PM

Ive got H100 going now, its caching latents

Aar.1 what speed where you getting ?

Furkan Gözükara SECourses•10/20/24, 1:55 PM

let me check video part

Furkan Gözükara SECourses•10/20/24, 1:55 PM

i am still editing :d

Furkan Gözükara SECourses•10/20/24, 1:55 PM

21.21 second

Furkan Gözükara SECourses•10/20/24, 1:55 PM

batch size 7

Furkan Gözükara SECourses•10/20/24, 1:56 PM

by the way if dataset cant be divided into 7 exactly, you wont get very accurate speed

FFurkan Gözükara SECourses 21.21 second

ar.1•10/20/24, 2:03 PM

so H100, batch size 12 I getting 16s/it

ar.1•10/20/24, 2:04 PM

65GB GPU memory usage

Aar.1 so H100, batch size 12 I getting 16s/it

Furkan Gözükara SECourses•10/20/24, 2:08 PM

wow nice

Furkan Gözükara SECourses•10/20/24, 2:08 PM

almost 1 second / it per image

ar.1•10/20/24, 2:09 PM

I can probably fit 14 batch size

FFurkan Gözükara SECourses almost 1 second / it per image

ar.1•10/20/24, 2:12 PM

so with H100 at $3 per hour, 10k images with 15 epochs (batch size 12) is gonna cost around $170

Aar.1 so with H100 at $3 per hour, 10k images with 15 epochs (batch size 12) is gonna ...

Furkan Gözükara SECourses•10/20/24, 2:29 PM

looks like way expensive compared to A6000

FFurkan Gözükara SECourses looks like way expensive compared to A6000

ar.1•10/20/24, 2:30 PM

yeah thats what im thinking

Furkan Gözükara SECourses•10/20/24, 2:30 PM

10k images 15 epochs would take 53$

Furkan Gözükara SECourses•10/20/24, 2:30 PM

0.31 cent per hour

ar.1•10/20/24, 2:30 PM

much better

Furkan Gözükara SECourses•10/20/24, 2:31 PM

yep

FFurkan Gözükara SECourses yep

ar.1•10/20/24, 2:32 PM

we cant do multi gpu with A6000 dreambooth finetuning right?

ar.1•10/20/24, 2:33 PM

I wish there was option to add more SSD in massed compute

ar.1•10/20/24, 2:34 PM

256GB on the A6000 isnt that great

Aar.1 we cant do multi gpu with A6000 dreambooth finetuning right?

Furkan Gözükara SECourses•10/20/24, 2:34 PM

yes you need 80 gb gpus

Aar.1 I wish there was option to add more SSD in massed compute

VinchOP•10/20/24, 2:34 PM

I keep having to upload checkpoints and delete them to make room for the next checkpoint

VVinch I keep having to upload checkpoints and delete them to make room for the next ch...

ar.1•10/20/24, 2:34 PM

yeah its a pain haha

ar.1•10/20/24, 2:34 PM

Has anyone tried A100

ar.1•10/20/24, 2:35 PM

I wonder its speed against the H100 as its almost half the price

VVinch I keep having to upload checkpoints and delete them to make room for the next ch...

Furkan Gözükara SECourses•10/20/24, 2:39 PM

you can delete existing apps and models

Furkan Gözükara SECourses•10/20/24, 2:39 PM

shown in new coming video

Furkan Gözükara SECourses•10/20/24, 2:39 PM

also yes i agree 256 GB disk space is very limiting for flux fine tuning :/

FFurkan Gözükara SECourses shown in new coming video

ar.1•10/20/24, 2:40 PM

very eager to watch it, whats the ETA ?

dsienra•10/20/24, 2:40 PM

@Dr. Furkan Gözükara this is cool!!!

Aar.1 very eager to watch it, whats the ETA ?

Furkan Gözükara SECourses•10/20/24, 2:40 PM

still editing

Ddsienra @Dr. Furkan Gözükara this is cool!!!

Furkan Gözükara SECourses•10/20/24, 2:40 PM

this is next hopefully after i published video to test

dsienra•10/20/24, 2:42 PM

sounds very promising!!!

snowmonkey•10/20/24, 2:51 PM

Does anyone have experience creating serverless Comfy API endpoint on RunPod ?
I keep getting "Max retries reached while waiting for image generation" with fresh installation of https://github.com/blib-la/runpod-worker-comfy

dsienra•10/20/24, 4:44 PM

@Dr. Furkan Gözükara Update https://huggingface.co/nyanko7/flux-dev-de-distill Lora training. 3 people same class in 1 lora with regularization, regularization captions just the class, lr 0.00005 unet and Te, very interesting results, high diversity on the class on inference on regular flux-dev, promoting for the trained subjects high resemblance, one curious thing a . on the prompt produces a total different image, for example same seed "Name class" different image than "Name class." is like the . works like a separation in the prompt, more to test, if you will use regularization use only the class for the captions, on previous tests I used captioned regularization and it produces unpredictable and unconcistent results. more things to test

nyanko7/flux-dev-de-distill · Hugging Face