Software Engineering Courses (SECourses)•13mo ago

but there is no fundamental problem on training with mixed images sizes in a set?

Dazzastrous [4090]•12/14/24, 4:21 PM

Could have used your airship Doc from Cog

cyberbol•12/14/24, 4:35 PM

What is it new option ReActor ?

TimsonOP•12/14/24, 6:14 PM

Is it normal that saving flux dev fp16 checkpoint takes 6 minutes?
I've tested the drive on that remote to have above 500MB/s writing speed, I don't see any CPU or GPU load during this operation, what could be the bottleneck?

TimsonOP•12/14/24, 6:14 PM

what is the "normal" duration of this operation?

Latentnaut•12/14/24, 6:35 PM

Hello group, do you think a finetune training of this product will capture all these little letters and details?

TTimson Is it normal that saving flux dev fp16 checkpoint takes 6 minutes? I've tested t...

TimsonOP•12/14/24, 6:45 PM

figured this out, I cheeped out on 16GB RAM + 16 GB swap that gets 99% filled during checkpoint save. Lesson learned, seems you need 32GB RAM for seamless training :heher:

Nneo do we got any alternatives like Massed Compute? other than runpod

Renzo•12/14/24, 7:38 PM

I have used https://www.hyperstack.cloud/. They have reasonable prices. But you have to configure everything mostly from scratch.