How to embrace variation and accept uncertainty in linguistic and psycholinguistic data analysis

Abstract

The use of statistical inference in linguistics and related areas like psychology typically involves a binary decision: either reject or accept some null hypothesis using statistical significance testing. When statistical power is low, this frequentist data-analytic approach breaks down: null results are uninformative, and effect size estimates associated with significant results are overestimated. Using an example from psycholinguistics, several alternative approaches are demonstrated for reporting inconsistencies between the data and a theoretical prediction. The key here is to focus on committing to a falsifiable prediction, on quantifying uncertainty statistically, and learning to accept the fact that – in almost all practical data analysis situations – we can only draw uncertain conclusions from data, regardless of whether we manage to obtain statistical significance or not. A focus on uncertainty quantification is likely to lead to fewer excessively bold claims that, on closer investigation, may turn out to be not supported by the data.

Conscious rereading is confirmatory: Evidence from bidirectional self-paced reading

“…Rather than using a binary criterion to distinguish "present" from "absent" effects, we focus on quantifying uncertainty by presenting the estimates from our analyses with their accompanying uncertainty intervals (Vasishth & Gelman, 2021). When interpreting the results, the reader should consider the width of the credible intervals, and whether the effect is consistent across the two experiments.…”

Sample Size Determination for Bayesian Hierarchical Models Commonly Used in Psycholinguistics

“…As statisticians have repeatedly pointed out (e.g., Wasserstein & Lazar, 2016;McShane et al, 2019), the goal certainly should not be quick binary conclusions based on oversimplified models. Uncertainty quantification is key to understanding data (Vasishth & Gelman, 2021), and as the van Doorn et al (2021) article also suggests, hierarchical models are a very important tool for achieving this goal.…”

The role of underspecification in relative clause attachment: Speed-accuracy tradeoff evidence.

“…While this finding demonstrates that reading is indeed influenced by task demands, it does not entail that parsing depends on task demands, or that readers underspecify. One piece of evidence to that end is a reanalysis of the Swets et al data by Vasishth (2021), which shows that Swets et al’s key finding, an interaction between question type and the effect of RC attachment, is significant only if untransformed reading times are analyzed with an analysis of variance (ANOVA). In an ANOVA analysis of log-reading time, or an analysis with linear mixed effects models (untransformed or log-transformed) the interaction fails to reach significance.…”