This idea doesn't exist in a void. Individual instances of subjective, conscious experience are called "qualia" in cognitive science and philosophy.

A lot of work, including a lot of experiments, has been done on this, and though of course it's impossible to completely prove anything, it seems very likely that humans with similar perceptual capacities do all have similar qualia.

The cognitive scientists have looked at the many humans who do have objectively different qualia due to measurable physiological conditions: color — blind people are a very good example. Many times, they themselves do not realize there is any impairment, but it's nearly trivial for us to objectively show that there is, and to prove it to the individuals themselves.

Separately, philosophers ask this question: if two people report having the same experience, if measuring the inputs to sensory organs and through to their brains show they have “the same” stimuli, and if no experiment can tell the difference, does it even have any meaning to say that their experiences are “actually” different? What exactly is this “difference” and where does it reside?

Sure, in some sense you can't really "know", but then you don't know your room isn't entirely different when your back is turned. At some point, you have to assume it isn't all an elaborate hoax, or else you might as well just throw up your hands and go home.