Posted: Sep 29, 2014 2:57 pm
Nice article indeed. In the wake of the fact that the '7σ detection' of primordial polarization from BICEP2 may now turn out to be galactic dust, I read this article about the pitfalls of p-values:
http://telescoper.wordpress.com/2013/11/12/the-curse-of-p-values/
Statistics is a bitch.
http://telescoper.wordpress.com/2013/11/12/the-curse-of-p-values/
The p-value merely specifies the probability that you would reject the null-hypothesis if it were correct. This is what you would call making a Type I error. It says nothing at all about the probability that the null hypothesis is actually a correct description of the data. To make that sort of statement you would need to specify an alternative distribution, calculate the distribution based on it, and hence determine the statistical power of the test, i.e. the probability that you would actually reject the null hypothesis when it is correct. To fail to reject the null hypothesis when it’s actually incorrect is to make a Type II error.
The Nature story mentioned above argues that in fact that results quoted with a p-value of 0.05 turn out to be wrong about 25% of the time. There are a number of reasons why this could be the case, including that the p-value is being calculated incorrectly, perhaps because some assumption or other turns out not to be true; a widespread example is assuming that the variates concerned are normally distributed. Unquestioning application of off-the-shelf statistical methods in inappropriate situations is a serious problem in many disciplines, but is particularly prevalent in the social sciences when samples are typically rather small.
Statistics is a bitch.