Failing to Disprove != Proving
Professor Volokh links to a study that finds no effect from abstinence education. This sparks a predictable argument in his comments section over sexual morality and public education, with both sides assuming that this study “proves” that abstinence education doesn’t work. It makes me wish that more people had a basic statistics education.
I’m always confused by a study that loudly announces “no difference” as the result of an experiment, as if that were a meaningful result. In the experimental design class that I had to take to major in Psychology, I learned that the point of an experiment is to disprove the null hypothesis, which assumes that there is no meaningful difference between the groups.
If you disprove the null hypothesis, then you really have something. You’ve demonstrated that the differences between the groups are probably due to something more than random noise. But if you fail to disprove the null hypothesis, then you’ve failed to prove anything. It’s really, really easy to fail to disprove the null hypothesis: Just run a lousy experiment. If you have a lousy design or a lousy method, then you’re almost guaranteed to fail to reject the null hypothesis.
I saw this firsthand in that Experimental Design class, because it required us to design and conduct our own experiment. Well over half the class could not disprove the null hypothesis, and it usually wasn’t because there was no difference between the groups. It was because they allowed too much random noise to seep into their results.
Suppose that you want to determine whether noise A is different from noise B. So you line up some subjects and have them listen to the noises. And none of the subjects can tell a difference. Does that prove that there’s no difference between A and B? No! There are plenty of alternative explanations: 1) Your subjects are deaf. 2) You conducted your experiment outside in a windstorm. 3) You unconsciously pressured your subjects not to notice the difference because that’s the result you wanted.
To summarize, as my Psych prof loved to repeat: Failing to disprove the null hypothesis does not prove the null hypothesis.