Disable Auto-Fullscreen Function on Steam Games

Good day,

I'm trying to use a discord chat overlay while playing games. To do this, I need to disable the Fullscreen option so the overlay can work properly.

Is there anyway to disable this auto fullscreen window action so that there is no need to keep disabling it whenever I launch a game.

Thanks.