In case anyone was wondering, I started a finetune on Vast using an A100 SXM4 (80GB VRAM) and it's d
In case anyone was wondering, I started a finetune on Vast using an A100 SXM4 (80GB VRAM) and it's doing ~3.16 sec/iteration.
MIt should be noted that on Vast.ai you can get this machine with 250 GB semi-permanent hd space for $0.78/h, which is pretty cheap…
Fhow many gpus
MThat was with one; now I'm testing 2x
Fi am still not able to get good fine tuning results with 10400 images
Fnice
FSXM4 difference
MI thought you were getting better times with the PCIE though, just under 3 seconds
MStill, decent value for the money
MOh BTW: using your heaviest, tier 1 config for the finetune
Fno 2x gpu gets way higher
Fbest was 4.1 second it
F2x a100
Fthe problem is fine tuning killing the model with big number of steps :d
Fi am testing 7 batch size now but it looks like didnt fix
Fi will try way lower LR
MOk with 2 GPUs I'm getting a stable 3.36 sec/it. I can never remember, though--I'm seeing:
steps: 0%| | 50/17500 [02:47<16:16:46, 3.36s/it, avr_loss=0.443]
steps: 0%| | 50/17500 [02:47<16:16:58, 3.36s/it, avr_loss=0.447]
(step #50 being done twice)
What does this mean? Will the final steps be 17500 or 35000?
steps: 0%| | 50/17500 [02:47<16:16:46, 3.36s/it, avr_loss=0.443]
steps: 0%| | 50/17500 [02:47<16:16:58, 3.36s/it, avr_loss=0.447]
(step #50 being done twice)
What does this mean? Will the final steps be 17500 or 35000?
MThis setup is $1.521/h with 326GB hard disk BTW
Fyou need to reduce your epoch by half
Fsince it clones training
Fwow this is cheap
Fdisable fused back pass it improves speed
MConfirmed, dropped to 3.14 sec/it. Now it's actually linearly increased
M2x the speed
MI'm not affiliated in any way, shape or form with Vast.ai, but they have great and friendly support (Massed has this, RunPod doesn't) and there's this Arizona datacenter that has a bunch of fine machines running for very little money. Good Internet speed, as well…
M(they offer GPUs from various providers, this Arizona one is particularly good)
Fye their A100 SXM4 is really good price
Ftheir A6000 more expensive than Massed Compute though
Fhow they can give this good price A100 SXM4 interesting
Fit is 1.89 usd on runpod
MI noticed that, it's crazy
M
MThis method is really pretty hard on the GPU BTW
MThe only machine I like on RunPod is the A40 for $0.39 (48GB VRAM), it's not spectacular, but it's sooo cheap.
Fit is worse than a6000
Fmassed compute is 31 cents :d
Falso runpod hdd and internet speed is like usually 1/10 for such gpus
MIndeed, but it's not always easy to find A6000s on MC, also thanks to your discount 
Ftrue i hope they add more
MHello @Furkan Gözükara SECourses , I just signed up for your patreon and downloaded these configs in the Best_Configs_v3.zip, a few questions 1) Are these configs for flux dreambooth? 2) What do the naming conventions mean for the different parts of the file name? 3) Which quality is better between 1 and 2? 4) Have you done a tutorial on Flux Dreambooth yet? 5) I guess it says to message you for a Patreon role. Thank you! 
Dfor Flux lora, when captioning a dataset, should I mention their defined characteristics? so if the subject is famously tall/pale skinned/muscular/whatever should those traits be mentioned or purposefully left out?
MMy impression when training for a specific person (ohwx, class woman) is that your top config has super-low effective learning rate. I generated samples with the same prompt and seed, and I can barely distinguish them between step 1000 and step 11,000. Is that something you noticed too? Should I not have kept the seed fixed?
Fthese are dreambooth
Fall quality 1 are same
Fthey are different vram and speed
Fwhat is your gpu
M4090
Ffor dreambooth i plan to record video very soon
Fi am doing few experiments
F4090 use 23100 mb
Fbut ensure that you have such free vram 