Inferential statistics as descriptive statistics: there is no replication crisis if we don't expect replication

Abstract

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced.

Exploratory behavior, but not aggressiveness, is correlated with breeding dispersal propensity in the highly philopatric thorn‐tailed rayadito

“…Fixed‐ and mixed‐effects models were fitted using the packages lme4 (Bates et al ) and MASS (Venables and Ripley ), while the nlme package (Pinheiro et al ) was used for the multivariate multilevel analysis. Continuous p‐values were reported as measure of compatibility between the null hypothesis and our data (Amrhein et al , Greenland , Wasserstein et al ).…”

The Continued Prevalence of Dichotomous Inferences at CHI

“…By dichotomous inference we refer to the classification of statistical evidence as either sufficient or insufficient, typically through the use of conventional cutoffs. Although dichotomous inference can be carried out using a variety of statistical methods (e.g., Bayes factors [21], posterior probabilities [2],...), the most popular procedure is by far null hypothesis significance testing (NHST). Thus this paper focuses on NHST.…”

Variation in interrogative adverbials: 'cuán', 'qué tan', 'cómo de','lo que' and 'lo'+adj./adv.+'que'

“…Recently, many voices have been raised against the use of null hypothesis testing to blindly dichotomize variables into those that do and do not influence the results. The idea that the alpha level of 0.05 is special or ideal has been criticized (Amhrein, Greenland & McShane 2019;Amrhein, Trafimow, andGreenland 2019, Cohen 1994). Kline (2004), on his part, actually suggests that an alpha level of 0.1 would be more appropriate in the behavioral sciences.…”