The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

Abstract

Words that are more surprising given context take longer to process. However, no incremental parsing algorithm has been shown to directly predict this phenomenon. In this work, we focus on a class of algorithms whose runtime does naturally scale in surprisal—those that involve repeatedly sampling from the prior. Our first contribution is to show that simple examples of such algorithms predict runtime to increase superlinearly with surprisal, and also predict variance in runtime to increase. These two predictions stand in contrast with literature on surprisal theory [Hale, 2001; Levy, 2008a], which assumes that the expected processing cost increases linearly with surprisal, and makes no prediction about variance. In the second part of this paper, we conduct an empirical study of the relationship between surprisal and reading time, using a collection of modern language models to estimate surprisal. We find that with better language models, reading time increases superlinearly in surprisal, and also that variance increases. These results are consistent with the predictions of sampling-based algorithms.

Word Frequency and Predictability Dissociate in Naturalistic Reading

“…As stressed in the Introduction , this result does not necessarily favor wholesale adoption of the procedural view that variation in word-by-word processing demand is driven primarily by representation-building rather than probabilistic inference. Indeed, considerable evidence has accumulated that probabilistic inference over sentence interpretations is a major component of language comprehension and is primarily responsible for effects of predictability on processing demand (Hoover et al, 2023 ; Meister et al, 2021 ; Shain et al, in press ; Smith & Levy, 2013 ; Szewczyk & Federmeier, 2022 ; Wilcox et al, 2020 , 2023 ). The frequency-predictability dissociation reported this this study may therefore highlight an important determinant of processing demand that is not explained by inference over sentence interpretations, but is consistent with distinct processes of memory retrieval.…”

“…In other words, although prior evidence favors an inferential (rather than a procedural preactivation-based) interpretation of predictability effects and thus implicates inference as a key “causal bottleneck” on processing demand, the present finding of surprisal-independent frequency effects could suggest limits on the scope of this bottleneck: frequency (and thus plausibly lexical retrieval) also plays a large and surprisal-independent role in determining how long participants spend reading words. Given the remarkable success of surprisal in accounting for a range of language processing phenomena across diverse experimental measures (Demberg & Keller, 2008 ; Frank & Bod, 2011 ; Frank et al, 2015 ; Heilbron et al, 2022 ; Hoover et al, 2023 ; Lopopolo et al, 2017 ; Roark et al, 2009 ; Shain et al, 2020 , in press ; Smith & Levy, 2013 ; van Schijndel & Schuler, 2015 ; Wilcox et al, 2020 ), discoveries highlighting the explanatory limits of surprisal offer opportunities for new insights into the mechanisms and representational format of incremental meaning construction during language comprehension (e.g., Huang et al, 2023 ; van Schijndel & Linzen, 2021 ).…”

“…GPT-2 has been shown to correlate strongly with human measures of language processing, both behavioral (Wilcox et al, 2020 ) and neural (Schrimpf et al, 2021 ), and GPT-2 surprisal outperforms cloze estimates as a predictor of human reading latencies (Shain et al, in press ). Based on strong prior evidence that predictability effects in reading are at least logarithmic (Hoover et al, 2023 ; Shain et al, in press ; Smith & Levy, 2013 ; Wilcox et al, 2020 , 2023 ; though cf. Brothers & Kuperberg, 2021 ), predictability is represented on a surprisal (negative log probability) scale.…”

“…First, unlike prior work, predictability is estimated using GPT-2, an incremental neural network language model (Radford et al, 2019 ) based on the transformer architecture (Vaswani et al, 2017 ). GPT-2 has been shown to strongly correlate with human measures of language processing, both in behavior (Hoover et al, 2023 ; Shain et al, in press ; Wilcox et al, 2020 ) and the brain (Jain et al, 2023 ; Schrimpf et al, 2021 ; Shain et al, 2022 ; Tuckute et al, 2024 ). In fact, one recent study showed that GPT-2 surprisal significantly outperformed cloze surprisal in predicting eye movements in the Provo eye-tracking dataset (Shain et al, in press ).…”

Large-scale evidence for logarithmic effects of word predictability on reading time

“…But what cognitive processes do predictability effects reflect? The answer to this question is tied to a major open debate about the cognitive architecture of human language comprehension ( 1 , 3 , 13 – 15 ).…”