signInWithWeb3 does not work on hardware wallets

I'm running into an issue where authenticating using the built in signInWithWeb3 function doesnt work for hardware wallets.

The issue is that ledger does not support 'sign in with solana' (aka off-chain message signing).

A big part of my userbase is using hardware wallets, so i need to find a solution to this...