Software Engineering Courses (SECourses)•16mo ago

Is massed compute the best way to train models? I have a 16GB memory M2 macbook. Last model took 33

Is massed compute the best way to train models? I have a 16GB memory M2 macbook. Last model took 33 hours to train so I must not have optimized. And I rented two A6000 GPUs.

Dylan NormandinOP•9/20/24, 1:12 AM

Also, why might this photo be blurry? I just used normal flux model, no fine tuning or LoRAs.

DDylan Normandin Also, why might this photo be blurry? I just used normal flux model, no fine tun...

CNOC•9/20/24, 1:15 AM

Change your CFG scale to 1 for flux

Dylan NormandinOP•9/20/24, 1:17 AM

awesome. tahnk you.

DDylan Normandin Is massed compute the best way to train models? I have a 16GB memory M2 macbook....

Furkan Gözükara SECourses•9/20/24, 1:34 AM

yep best

DDylan Normandin Also, why might this photo be blurry? I just used normal flux model, no fine tun...

Furkan Gözükara SECourses•9/20/24, 1:35 AM

happens sometimes

Furkan Gözükara SECourses•9/20/24, 1:35 AM

make more steps

Furkan Gözükara SECourses•9/20/24, 1:35 AM

and cfg scale 1

ministryofspaceandtime•9/20/24, 12:59 PM

Hi there!
Attempting to fully train FLUX with dreambooth on runpod.
Does anybody encounter same trouble?

[rank1]: ValueError: DistributedDataParallel device_ids and output_device arguments only work with single-device/multiple-device GPU modules or CPU modules, but got device_ids [1], output_device 1, and module parameters {device(type='cpu')}.

While using multiple GPU getting this error. Using 2 A100 PCle. If turning off checkbox multi GPU and switching Number of processes to 1 working great.

Using Quality_1_21820MB_16_12_Second_IT.json from Best_Configs_v3

message.txt5.28KB

dxqb•9/20/24, 2:48 PM

it's quite difficult to find the right LR for Flux. Because they all work

Mministryofspaceandtime Hi there! Attempting to fully train FLUX with dreambooth on runpod. Does anybod...

druhl•9/20/24, 3:58 PM

Yes, I also got to know recently (and after planning to start one training), that multiple GPUs won't work on dreambooth training x flux yet. I shoulda read the post better lol.

Ddruhl Yes, I also got to know recently (and after planning to start one training), tha...

ministryofspaceandtime•9/20/24, 9:04 PM

Gosh, am I arrogant? I just skimmed through it.

Mministryofspaceandtime Hi there! Attempting to fully train FLUX with dreambooth on runpod. Does anybod...

Furkan Gözükara SECourses•9/20/24, 9:12 PM

hi welcome

Furkan Gözükara SECourses•9/20/24, 9:12 PM

i trained lora on multi gpu

Furkan Gözükara SECourses•9/20/24, 9:12 PM

fine tuning had failed on 48 gb gpu

Furkan Gözükara SECourses•9/20/24, 9:12 PM

but didnt try on 80 yet

Furkan Gözükara SECourses•9/20/24, 9:13 PM

but i didnt get thgis error on massed compute

Furkan Gözükara SECourses•9/20/24, 9:13 PM

it was OOM

Furkan Gözükara SECourses•9/20/24, 9:14 PM

are you using my config @ministryofspaceandtime

FFurkan Gözükara SECourses are you using my config <@668488071380992050>

ministryofspaceandtime•9/21/24, 10:17 AM

Yes, was trying this configs. Every one of them works fine on single GPU and failed on multiple. (by the way I'm using RunPod, not Massed Compute)

Mministryofspaceandtime Yes, was trying this configs. Every one of them works fine on single GPU and fa...

Furkan Gözükara SECourses•9/21/24, 11:36 AM

i will try on runpod lets see

Furkan Gözükara SECourses•9/21/24, 12:05 PM

installing right now on 2x a100

Mministryofspaceandtime Yes, was trying this configs. Every one of them works fine on single GPU and fa...

Furkan Gözükara SECourses•9/21/24, 12:18 PM

itm is working

Furkan Gözükara SECourses•9/21/24, 12:18 PM

Furkan Gözükara SECourses•9/21/24, 12:19 PM

Quality_1_27500MB_6_26_Second_IT.json

Furkan Gözükara SECourses•9/21/24, 12:20 PM

but it is ridiciliously slow slower than 1 gpu

Prototype1072•9/22/24, 6:39 AM

hey guys, how can i fix this, its been eating my head for a while now. Any help would be appreciated

PPrototype1072 hey guys, how can i fix this, its been eating my head for a while now. Any help ...

Furkan Gözükara SECourses•9/22/24, 3:31 PM

what are you trying to do?

macba•9/22/24, 4:00 PM

Hi there! @Furkan Gözükara SECourses , I read your latest greatest post about multiple-GPU Dreambooth finetuning for Flux. Please tell me if I got this right: if you want to use multiple GPUs, they must each have ≥80 GB RAM (so, top-of-the-line A100's or H100's). Still, you don't get linear decreases in processing time, at least with A100's (with 2x A100's you don't get 2x speed). Thus, for that money (at least on RunPod), you're better off with one L40S.

However, what about 4x A6000's? Those don't have 80 GB VRAM. Are they still cost-effective? EDIT: referring to Massed Compute's 4x A6000's at half-price with your special code, as well as those at full price.