Software Engineering Courses (SECourses)•16mo ago

hey guys, how can i fix this, its been eating my head for a while now. Any help would be appreciated

PPrototype1072 hey guys, how can i fix this, its been eating my head for a while now. Any help ...

Furkan Gözükara SECourses•9/22/24, 3:31 PM

what are you trying to do?

macba•9/22/24, 4:00 PM

Hi there! @Furkan Gözükara SECourses , I read your latest greatest post about multiple-GPU Dreambooth finetuning for Flux. Please tell me if I got this right: if you want to use multiple GPUs, they must each have ≥80 GB RAM (so, top-of-the-line A100's or H100's). Still, you don't get linear decreases in processing time, at least with A100's (with 2x A100's you don't get 2x speed). Thus, for that money (at least on RunPod), you're better off with one L40S.

However, what about 4x A6000's? Those don't have 80 GB VRAM. Are they still cost-effective? EDIT: referring to Massed Compute's 4x A6000's at half-price with your special code, as well as those at full price.

Mmacba Hi there! <@205854764540362752> , I read your latest greatest post about multipl...

Furkan Gözükara SECourses•9/22/24, 5:16 PM

it doesnt fit into a6000

Furkan Gözükara SECourses•9/22/24, 5:16 PM

that is the problem

Furkan Gözükara SECourses•9/22/24, 5:16 PM

from 27 gb to jumps 75 gb vram

Furkan Gözükara SECourses•9/22/24, 5:16 PM

when doing multi gpu

Furkan Gözükara SECourses•9/22/24, 5:16 PM

i reported this to kohya but he thinks it is accurate

macba•9/22/24, 5:18 PM

So, using multiple GPUs gives marginal, non-cost-effective advantage only when using ≥80GB GPUs, right?

Mmacba So, using multiple GPUs gives marginal, non-cost-effective advantage only when u...

Furkan Gözükara SECourses•9/22/24, 5:21 PM

at least for 2x

Furkan Gözükara SECourses•9/22/24, 5:22 PM

also these are pciex

Furkan Gözükara SECourses•9/22/24, 5:22 PM

sxm maybe different

Jjkgnl I used a Saas solution where I trained on the same 20 photos using Flux Lora, it...

sexyboi•9/22/24, 7:24 PM

What saas service did you use that was able to do it in only 30 minutes? I'd love to give that a shot

FFurkan Gözükara SECourses sxm maybe different

macba•9/22/24, 9:13 PM

In case anyone was wondering, I started a finetune on Vast using an A100 SXM4 (80GB VRAM) and it's doing ~3.16 sec/iteration.

Mmacba In case anyone was wondering, I started a finetune on Vast using an A100 SXM4 (8...

macba•9/22/24, 9:15 PM

It should be noted that on Vast.ai you can get this machine with 250 GB semi-permanent hd space for $0.78/h, which is pretty cheap…

Mmacba In case anyone was wondering, I started a finetune on Vast using an A100 SXM4 (8...

Furkan Gözükara SECourses•9/22/24, 9:26 PM

how many gpus

FFurkan Gözükara SECourses how many gpus

macba•9/22/24, 9:26 PM

That was with one; now I'm testing 2x

Furkan Gözükara SECourses•9/22/24, 9:26 PM

i am still not able to get good fine tuning results with 10400 images

Mmacba That was with one; now I'm testing 2x

Furkan Gözükara SECourses•9/22/24, 9:26 PM

nice

Furkan Gözükara SECourses•9/22/24, 9:26 PM

SXM4 difference

macba•9/22/24, 9:27 PM

I thought you were getting better times with the PCIE though, just under 3 seconds

macba•9/22/24, 9:27 PM

Still, decent value for the money

macba•9/22/24, 9:28 PM

Oh BTW: using your heaviest, tier 1 config for the finetune

Mmacba I thought you were getting better times with the PCIE though, just under 3 secon...

Furkan Gözükara SECourses•9/22/24, 9:30 PM

no 2x gpu gets way higher

Furkan Gözükara SECourses•9/22/24, 9:30 PM

best was 4.1 second it

Furkan Gözükara SECourses•9/22/24, 9:31 PM

2x a100

Furkan Gözükara SECourses•9/22/24, 9:31 PM

the problem is fine tuning killing the model with big number of steps :d

Furkan Gözükara SECourses•9/22/24, 9:31 PM

i am testing 7 batch size now but it looks like didnt fix

Furkan Gözükara SECourses•9/22/24, 9:31 PM

i will try way lower LR

FFurkan Gözükara SECourses best was 4.1 second it

macba•9/22/24, 9:46 PM

Ok with 2 GPUs I'm getting a stable 3.36 sec/it. I can never remember, though--I'm seeing:

steps: 0%| | 50/17500 [02:47<16:16:46, 3.36s/it, avr_loss=0.443]
steps: 0%| | 50/17500 [02:47<16:16:58, 3.36s/it, avr_loss=0.447]

(step #50 being done twice)

What does this mean? Will the final steps be 17500 or 35000?

Mmacba Ok with 2 GPUs I'm getting a stable 3.36 sec/it. I can never remember, though--I...

macba•9/22/24, 9:47 PM

This setup is $1.521/h with 326GB hard disk BTW

Mmacba Ok with 2 GPUs I'm getting a stable 3.36 sec/it. I can never remember, though--I...

Furkan Gözükara SECourses•9/22/24, 9:49 PM

you need to reduce your epoch by half

Furkan Gözükara SECourses•9/22/24, 9:49 PM

since it clones training

Mmacba This setup is $1.521/h with 326GB hard disk BTW

Furkan Gözükara SECourses•9/22/24, 9:50 PM

wow this is cheap

Furkan Gözükara SECourses•9/22/24, 9:50 PM

disable fused back pass it improves speed

macba•9/22/24, 9:54 PM

Confirmed, dropped to 3.14 sec/it. Now it's actually linearly increased

macba•9/22/24, 9:55 PM

2x the speed

macba•9/22/24, 9:56 PM

I'm not affiliated in any way, shape or form with Vast.ai, but they have great and friendly support (Massed has this, RunPod doesn't) and there's this Arizona datacenter that has a bunch of fine machines running for very little money. Good Internet speed, as well…

macba•9/22/24, 9:57 PM

(they offer GPUs from various providers, this Arizona one is particularly good)

Furkan Gözükara SECourses•9/22/24, 10:00 PM

ye their A100 SXM4 is really good price

Furkan Gözükara SECourses•9/22/24, 10:02 PM

their A6000 more expensive than Massed Compute though

Furkan Gözükara SECourses•9/22/24, 10:02 PM

how they can give this good price A100 SXM4 interesting

Furkan Gözükara SECourses•9/22/24, 10:02 PM

it is 1.89 usd on runpod

macba•9/22/24, 10:06 PM

I noticed that, it's crazy