Statistical properties of the speed-accuracy trade-off (SAT) paradigm in sentence processing

Abstract

Studies of the speed-accuracy trade-off (SAT) have been influential in arguing for the direct-access model of retrieval in sentence processing. The direct-access model assumes that long-distance dependencies (like subject-verb agreement) rely on a content-addressable search for the correct representation in memory. In this paper, we address two important weaknesses in the statistical methods standardly used for analysing SAT data. First, these methods are based on non-hierarchical modelling. We show how a hierarchical model can be fit to SAT data, and we test parameter recovery in this more conservative model. The parameters most relevant to the direct-access account cannot be accurately estimated. This may be due to the nature of SAT data or the standard SAT model. Second, the power properties of SAT studies are unknown. We conduct a power analysis and show that inferences from null results to the null hypothesis, though commonplace in the SAT literature, are likely unwarranted.

The role of underspecification in relative clause attachment: Speed-accuracy tradeoff evidence.

“…In consequence, the model-comparison metric adjusted R 2 that is typically used in the analysis of SAT data places an undue emphasis on the part of the SATF with the highest sampling variability. The result is high variability of by-subject parameter estimates and relatively low statistical power even for modest effect sizes (Loga cev & Bozkurt, 2021; Pankratz et al, 2021). Hierarchical models (Gelman & Hill, 2006) with the participants' responses as a dependent variable can be used to estimate the sensitivity d 0 and the response criterion c (e.g., DeCarlo, 1998;Wright & London, 2009) while avoiding most of the above problems.…”

“…In consequence, the model-comparison metric adjusted R 2 that is typically used in the analysis of SAT data places an undue emphasis on the part of the SATF with the highest sampling variability. The result is high variability of by-subject parameter estimates and relatively low statistical power even for modest effect sizes (Logačev & Bozkurt, 2021; Pankratz et al, 2021).…”

Some Right Ways to Analyze (Psycho)Linguistic Data

“…As discussed in Vasishth & Gelman (2021), this has real, practical consequences for linguistics; if power is low, even if one repeatedly gets null results across multiple experiments, this does not imply that one has found evidence in favor of the null. The field is full of incorrect statistical inferences based on such null results from underpowered studies (e.g., Pankratz et al 2021;Logacev & Bozkurt 2021).…”