Chapter 6 Critical Discussion of Null Hypothesis Significance Testing

Key concepts: problems with null hypothesis significance testing, meta-analysis, replication, frequentist versus Bayesian inference, theoretical population, data generating process.

Watch this micro lecture on criticisms of null hypothesis significance testing for an overview of the chapter.

Summary

How important is null hypothesis significance testing?

In the preceding chapters, we learned to test null hypotheses. Null hypothesis significance testing is widely used in the social and behavioral sciences. There are, however, problems with null hypothesis significance tests that are increasingly being recognized.

The statistical significance of a null hypothesis test depends strongly on the size of the sample (Chapters 4 and 4.2.6), so non-significance may merely mean that the sample is too small. In contrast, irrelevant tiny effects can be statistically significant in a very large sample. Finally, we normally test a null hypothesis that there is no effect whereas we have good reasons to believe that there is an effect in the population. What does a significant test result really tell us if we reject an unlikely null hypothesis?

Among the alternatives to null hypothesis significance testing, using a confidence interval to estimate effects in the population is easiest to apply. It is closely related to null hypothesis testing, as we have seen in Section ??, but it offers us information with which we can draw a more nuanced conclusion about our results.