Yep, this is where your argument falls down, because you don't actually know the dynamic range of your signal for certain before the recording session.

So even though if you knew what your maximum amplitude was going to be before the session, 16-bit would be fine, you don't actually know this: so you need 24-bit to make sure you can set both the noise floor low enough, and the peak amplitude so it won't clip, and not have to worry about it.