When I'm screen recording a game, seems like the kernel scheduler puts the game in a priority and actually leaves screen recorder behind, making it so the recording is lagging, even though the game didn't lag for me and was running top speed.
I found a workaround, the issue is very specifically only when set to the primary monitor, so I just set the other monitor to primary, with some custom borderless window settings and it's good to go