are you using my config <@668488071380992050>
are you using my config @ministryofspaceandtime
MYes, was trying this configs. Every one of them works fine on single GPU and failed on multiple. (by the way I'm using RunPod, not Massed Compute)
Fi will try on runpod lets see
Finstalling right now on 2x a100
Fitm is working
F
FQuality_1_27500MB_6_26_Second_IT.json
Fbut it is ridiciliously slow slower than 1 gpu 
Phey guys, how can i fix this, its been eating my head for a while now. Any help would be appreciated
Fwhat are you trying to do?
MHi there! @Furkan Gözükara SECourses , I read your latest greatest post about multiple-GPU Dreambooth finetuning for Flux. Please tell me if I got this right: if you want to use multiple GPUs, they must each have ≥80 GB RAM (so, top-of-the-line A100's or H100's). Still, you don't get linear decreases in processing time, at least with A100's (with 2x A100's you don't get 2x speed). Thus, for that money (at least on RunPod), you're better off with one L40S.
However, what about 4x A6000's? Those don't have 80 GB VRAM. Are they still cost-effective? EDIT: referring to Massed Compute's 4x A6000's at half-price with your special code, as well as those at full price.
However, what about 4x A6000's? Those don't have 80 GB VRAM. Are they still cost-effective? EDIT: referring to Massed Compute's 4x A6000's at half-price with your special code, as well as those at full price.
Fit doesnt fit into a6000
Fthat is the problem
Ffrom 27 gb to jumps 75 gb vram
Fwhen doing multi gpu
Fi reported this to kohya but he thinks it is accurate
MSo, using multiple GPUs gives marginal, non-cost-effective advantage only when using ≥80GB GPUs, right?
Fat least for 2x
Falso these are pciex
Fsxm maybe different
SWhat saas service did you use that was able to do it in only 30 minutes? I'd love to give that a shot
MIn case anyone was wondering, I started a finetune on Vast using an A100 SXM4 (80GB VRAM) and it's doing ~3.16 sec/iteration.
MIt should be noted that on Vast.ai you can get this machine with 250 GB semi-permanent hd space for $0.78/h, which is pretty cheap…
Fhow many gpus
MThat was with one; now I'm testing 2x
Fi am still not able to get good fine tuning results with 10400 images
Fnice
FSXM4 difference
MI thought you were getting better times with the PCIE though, just under 3 seconds
MStill, decent value for the money
MOh BTW: using your heaviest, tier 1 config for the finetune
Fno 2x gpu gets way higher
Fbest was 4.1 second it
F2x a100
Fthe problem is fine tuning killing the model with big number of steps :d
Fi am testing 7 batch size now but it looks like didnt fix
Fi will try way lower LR
MOk with 2 GPUs I'm getting a stable 3.36 sec/it. I can never remember, though--I'm seeing:
steps: 0%| | 50/17500 [02:47<16:16:46, 3.36s/it, avr_loss=0.443]
steps: 0%| | 50/17500 [02:47<16:16:58, 3.36s/it, avr_loss=0.447]
(step #50 being done twice)
What does this mean? Will the final steps be 17500 or 35000?
steps: 0%| | 50/17500 [02:47<16:16:46, 3.36s/it, avr_loss=0.443]
steps: 0%| | 50/17500 [02:47<16:16:58, 3.36s/it, avr_loss=0.447]
(step #50 being done twice)
What does this mean? Will the final steps be 17500 or 35000?
MThis setup is $1.521/h with 326GB hard disk BTW
Fyou need to reduce your epoch by half
Fsince it clones training
Fwow this is cheap
Fdisable fused back pass it improves speed
MConfirmed, dropped to 3.14 sec/it. Now it's actually linearly increased
M2x the speed
MI'm not affiliated in any way, shape or form with Vast.ai, but they have great and friendly support (Massed has this, RunPod doesn't) and there's this Arizona datacenter that has a bunch of fine machines running for very little money. Good Internet speed, as well…
M(they offer GPUs from various providers, this Arizona one is particularly good)
Fye their A100 SXM4 is really good price
Ftheir A6000 more expensive than Massed Compute though
Fhow they can give this good price A100 SXM4 interesting
Fit is 1.89 usd on runpod