Software Engineering Courses (SECourses)•6mo ago

Have an 3090 and recently did a fresh install of Kohya, followed by 'Windows_RTX5000_Series_Upgrade_

Have an 3090 and recently did a fresh install of Kohya, followed by 'Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat', and previously my speed was ~8s/it but now its ~11s/it. Tried reverting and doing fresh installs again but the same issue. Any idea on what's the best way to debug what's causing this?

Gghstoic Have an 3090 and recently did a fresh install of Kohya, followed by 'Windows_RTX...

Furkan Gözükara SECourses•7/1/25, 12:38 PM

is your dataset all 1024x1024?

FFurkan Gözükara SECourses is your dataset all 1024x1024?

ghstoicOP•7/1/25, 12:40 PM

my actual images are larger, but my 'Max resolution' is set to 1024x1024 (so assuming they're getting resized to 1024x1024 themselves)

Gghstoic my actual images are larger, but my 'Max resolution' is set to 1024x1024 (so ass...

Furkan Gözükara SECourses•7/1/25, 12:40 PM

that is why

Furkan Gözükara SECourses•7/1/25, 12:40 PM

it wont resize to 1024x1024

Furkan Gözükara SECourses•7/1/25, 12:40 PM

it will only resize small side to 1024

ghstoicOP•7/1/25, 12:41 PM

sorry, I should have said that they're all 1:1 aspect ratio, so would end up being 1024x1024

Gghstoic sorry, I should have said that they're all 1:1 aspect ratio, so would end up bei...

Furkan Gözükara SECourses•7/1/25, 12:41 PM

how much watt your gpu using did you check?

Furkan Gözükara SECourses•7/1/25, 12:43 PM

i mean like this

Furkan Gözükara SECourses•7/1/25, 12:43 PM

ghstoicOP•7/1/25, 12:43 PM

just rerunning it now - 2 mins

ghstoicOP•7/1/25, 12:46 PM

Guessing I need to try this right: tips-info

ghstoicOP•7/1/25, 12:47 PM

^not sure if that is comfyui specific and not related to Kohya though

ghstoicOP•7/1/25, 12:49 PM

Gghstoic Click to see attachment

Furkan Gözükara SECourses•7/1/25, 12:58 PM

you need to increase block swap count

ghstoicOP•7/1/25, 1:26 PM

ah ok, I haven't changed the config at all from when it was running at 8s/it. I've already tried blocks to swap ranging from 12 - 22 in each of the 3 set-ups below, but doesn't seem to get it past 11s/it:

Old install + old config + 'Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat' file -> 11s/it
New install + old config -> 11s/it
New install + new config -> 11s/it
New install + old config + 'Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat' -> 11s/it
New install + new config + 'Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat' -> 11s/it

I can't seem to revert back to my old install when it was at 8s/it. I do have a backup of the kohya_ss folder, but copy + pasting it back over the new one doesn't seem to fix it either. I'm guessing one of my packages was permanent upgraded to when I ran the 'Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat' script that my RTX 3090 doesn't like? Not sure if this is strictly true (i.e. if packages are installed in their own venv each time) as this stuff is a little beyond me sometimes

Gghstoic ah ok, I haven't changed the config at all from when it was running at 8s/it. I'...

Furkan Gözükara SECourses•7/1/25, 1:41 PM

are you waiting at least 100 steps

Furkan Gözükara SECourses•7/1/25, 1:41 PM

to see real speed?

ghstoicOP•7/1/25, 1:43 PM

yup, i'm waiting for it to bottom out. Over 3000 steps from run from last night:

Gghstoic yup, i'm waiting for it to bottom out. Over 3000 steps from run from last night:

Furkan Gözükara SECourses•7/1/25, 1:44 PM

so even you change block swap

Furkan Gözükara SECourses•7/1/25, 1:45 PM

you cant get better?

Furkan Gözükara SECourses•7/1/25, 1:45 PM

your watt usage is low

Furkan Gözükara SECourses•7/1/25, 1:45 PM

Furkan Gözükara SECourses•7/1/25, 1:45 PM

let me test mine

Furkan Gözükara SECourses•7/1/25, 1:45 PM

this is dreambooth or lora?

ghstoicOP•7/1/25, 1:53 PM

dreambooth

FFurkan Gözükara SECourses so even you change block swap

ghstoicOP•7/1/25, 1:53 PM

yup, that's right

Cryptozow.x•7/1/25, 6:50 PM

Hey @Furkan Gözükara SECourses I hope you're doing great, just wanted to thank you for your latest tutorials and also ask if the video upscaler is still in the works coming soon, or you are now prioritizing other stuff (really hard to catch up with ai nowadays). After I saw your latest video upscaler new implementation posts I'm really looking forward to test it out.

Shyam Nimmana•7/1/25, 7:35 PM

For flux lora training. I have images 2046 * 3050px size but it needs 1024 * 1024px. What should I do?

Gghstoic ah ok, I haven't changed the config at all from when it was running at 8s/it. I'...

ghstoicOP•7/1/25, 7:42 PM

Any idea on this Furkan? I've just done another fresh install + latest cudnn + 'Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat' + Cuda 12.9. Running on a fresh windows reboot (so no background process), block swap set to 22, and still limited to 11.37s/it

really confused on what else I can try. Just odd how I was getting 8s/it beforehand and now with more up-to-date packages its running slower. I can't really revert it either, so would love to know what you'd try to do next in my situation