Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations

Abstract

Null hypothesis statistical significance tests (NHST) are widely used in quantitative research in the empirical sciences including scientometrics. Nevertheless, since their introduction nearly a century ago significance tests have been controversial. Many researchers are not aware of the numerous criticisms raised against NHST. As practiced, NHST has been characterized as a 'null ritual' that is overused and too often misapplied and misinterpreted. NHST is in fact a patchwork of two fundamentally different classical statistical testing models, often blended with some wishful quasi-Bayesian interpretations. This is undoubtedly a major reason why NHST is very often misunderstood. But NHST also has intrinsic logical problems and the epistemic range of the information provided by such tests is much more limited than most researchers recognize. In this article we introduce to the scientometric community the theoretical origins of NHST, which is mostly absent from standard statistical textbooks, and we discuss some of the most prevalent problems relating to the practice of NHST and trace these problems back to the mix-up of the two different theoretical origins. Finally, we illustrate some of the misunderstandings with examples from the scientometric literature and bring forward some modest recommendations for a more sound practice in quantitative data analysis.

Calculation of a distribution free estimate of effect size and confidence intervals using VBA/Excel

“…If the p-value is below a predefined, arbitrary threshold, usually p<0.05, the result is explained as evidence in favor of an alternative hypothesis, with smaller p-values taken as stronger evidence in favor of the alternative hypothesis. Importantly, p-values do not signify the strength of evidence in favor of an alternative hypothesis (Goodman, 2008;Schneider, 2015). Moreover, NHST and the resulting p-value do not give any information on the magnitude of the difference (Nakagawa and Cuthill, 2007;Cumming, 2014;Ivarsson et al, 2015;Motulsky, 2015).…”

Mapping international impact of Danish neuroscience from 2004 to 2015 using tailored scientometric methodology

“…We have not provided uncertainty estimates in relation to the indicators presented. The issue of quantifying uncertainty is a contested topic especially in relation to research evaluation and scientometric indicators (Schneider, 2013(Schneider, , 2015(Schneider, , 2016Hicks et al, 2015). A current pragmatic compromise, although still disputed, is to use so-called 'stability intervals', which are bootstrapped confidence intervals (Colliander & Ahlgren, 2011;Schneider & van Leeuwen, 2014).…”

Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

“…Consequently, the Type I error rate remains constant if researchers simply repeat the same test over and over again using different samples that have been randomly drawn from the exact same population. However, this first situation is somewhat hypothetical and may even be regarded as impossible in the social sciences because populations of people change over time and location (e.g., Gergen, 1973;Iso-Ahola, 2017;Schneider, 2015;Serlin, 1987;Stroebe & Strack, 2014). Yesterday's population of psychology undergraduate students from the University of Newcastle, Australia will be a different population to today's population of psychology undergraduate students from the University of Newcastle, Australia.…”