Remote play not working in Steam itself or with Moonlight unless HW en+decoding is disabled

Hopefully the title is clear enough, I couldn't fit the whole thing ahaha. Let me break it down more clearly though:

I can't do remote streams to my bazzite htpc from my main machine unless I disable hardware encoding on the host and disable hardware decoding on the Bazzite HTPC. Both are running NVIDIA cards, a RTX 4070 Super on the host and a RTX 2070 on the HTPC.

It's not ideal for performance to put all that on the CPU while also trying to run games at higher performance, of course. So I was wondering if anyone had any idea of what was happening and if there was any solution. Is there some kind of bizarre setting I'm missing or something?
Was this page helpful?