A hard wallet is just a really fancy shoebox.

The only difference between the hard wallet and a shoebox is that it’s harder to open a hard wallet. If someone steal your seed phrase and/or hard wallet, you lose access to your money. it doesn’t matter if the thief knows your passcode or pass phrase and can actually gain access to your funds or not.

I think having a pass phrase and multi copies of your seed phrase might be the best solution.