Lossy‐Context Surprisal: An Information‐Theoretic Model of Memory Effects in Sentence Processing

Futrell, Richard; Gibson, Edward; Lévy, Roger

doi:10.1111/cogs.12814

Cited by 122 publications

(152 citation statements)

References 127 publications

Supporting

Mentioning

145

Contrasting

Unclassified

Order By: Relevance

“…At a minimum, our results indicate that the relationship between surprisal and reading times is not linear in extreme conditions such as MV/RR garden path constructions. It is possible that such a non-linear relationship may arise naturally if surprisal is augmented with the noisy channel or lossy context hypotheses Bicknell and Levy, 2010;Gibson et al, 2013;Futrell et al, 2020). However, this possibility cannot explain the qualitatively incorrect reading time predictions we observed in NP/Z constructions.…”

Section: Discussionmentioning

confidence: 79%

Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty

Schijndel¹,

Linzen²

2020

Preprint

View full text Add to dashboard Cite

The disambiguation of a syntactically ambiguous sentence in favor of dispreferred parse can lead to slower reading at the disambiguation point. This phenomenon, referred to as a garden path effect, has motivated models in which readers only maintain a subset of the possible parses of the sentence; reverting to a discarded parse requires costly reanalysis. More recently, it has been proposed that the garden path effect can be reduced to surprisal arising in a fully parallel parser: words consistent with the initially dispreferred but ultimately correct parse are simply less predictable than those consistent with the incorrect parse. The surprisal account is more parsimonious since predictability has pervasive effects in reading far beyond garden path sentences. Crucially, this account predicts a linear effect of surprisal: the difficulty experienced by readers should be proportional to the difference in word surprisal between the ultimately correct and ultimately incorrect interpretations. To test this prediction, we estimated word-by-word surprisal using recurrent neural network language models, comparing those estimates to self-paced reading times for three garden path constructions. While surprisal successfully predicted the existence of garden path responses, it severely underpredicted the magnitude of all of the garden path effects. Further, the relative size of the predicted effects was inconsistent with the relative size of the responses in humans, indicating that a differently scaled linking function would not be able to predict the response magnitudes either. These results support two-stage processing models in which recovery mechanisms beyond predictability are involved in processing garden path sentences.

show abstract

Section: Discussionmentioning

confidence: 79%

Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty

Schijndel¹,

Linzen²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The interaction between past exposure and working memory is an active area of research. Recent extensions to the surprisal metric (Hale, 2001; Levy, 2008) such as the lossy‐context surprisal (Futrell et al, 2019; Futrell & Levy, 2017) have been shown to predict locality as well as the forgetting effects pattern in English and German. Certain models of working memory that can perform differential resource allocation based on prior experience (Daily, Lovett, & Reder, 2001; Lovett, Daily, & Reder, 2000) also hold promise in accounting for the interaction of limited working memory capacity and past exposure.…”

Section: Discussionmentioning

confidence: 99%

“…A better prediction maintenance capability in German allows for longer HD between nouns and verbs in the language. One way to formalize better maintenance is by equating prediction strength with memory activation, such that strong/better prediction of a linguistic unit (e.g., verb) leads to a higher memory activation of that linguistic entity and a weak prediction will lead to lower memory activation (Husain & Vasishth, 2014; Husain et al, 2014; also see Campanelli, Van Dyke, & Marton, 2018; Futrell, Gibson, & Levy, 2019). Alternatively, from a connectionist perspective, the network adapts its weights to the frequent word order pattern of the language, thereby becoming better at handling it.…”

Section: Introductionmentioning

confidence: 99%

Word Order Typology Interacts With Linguistic Complexity: A Cross‐Linguistic Corpus Study

et al. 2020

View full text Add to dashboard Cite

Much previous work has suggested that word order preferences across languages can be explained by the dependency distance minimization constraint (Ferrer-i Cancho, 2008Hawkins, 1994). Consistent with this claim, corpus studies have shown that the average distance between a head (e.g., verb) and its dependent (e.g., noun) tends to be short cross-linguistically show that the comprehension system can adapt to the typological properties of a language, for example, verb-final order, leading to more complex structures, for example, having longer linear distance between a head and its dependent. In this paper, we conduct a corpus study for a group of 38 languages, which were either Subject-Verb-Object (SVO) or Subject-Object-Verb (SOV), in order to investigate the role of word order typology in determining syntactic complexity. We present results aggregated across all dependency types, as well as for specific verbal (objects, indirect objects, and adjuncts) and nonverbal (nominal, adjectival, and adverbial) dependencies. The results suggest that dependency distance in a language is determined by the default word order of a language, and crucially, the direction of a dependency (whether the head precedes the dependent or follows it; e.g., whether the noun precedes the verb or follows it). Particularly we show that in SOV languages (e.g., Hindi, Korean) as well as SVO languages (e.g., English, Spanish), longer linear distance (measured as number of words) between head and dependent arises in structures when they mirror the default word order of the language. In addition to showing results on linear distance, we also investigate the influence of word order typology on hierarchical distance (HD; measured as number of heads between head and dependent). The results for HD are similar to that of linear distance. At the same time, in comparison to linear distance, the influence of adaptability on HD seems less strong. In particular, the results Correspondence should be sent to Himanshu Yadav is now affiliated with University of Potsdam, Germany. Vishakha Shukla is now affiliated with Shroff Charity Eye Hospital, Delhi, India. show that most languages tend to avoid greater structural depth. Together, these results show evidence for "limited adaptability" to the default word order preferences in a language. Our results support a large body of work in the processing literature that highlights the importance of linguistic exposure and its interaction with working memory constraints in determining sentence complexity. Our results also point to the possible role of other factors such as the morphological richness of a language and a multifactor account of sentence complexity remains a promising area for future investigation.

show abstract

“…However, to date no such explanation is available for all of the processing patterns summarized in Section 3.1. See Lewis et al (2006, the last bullet in Box 3), Campanelli, Van Dyke, and Marton (2018), and Futrell, Gibson, and Levy (2020), inter alia .…”

mentioning

confidence: 99%

Quantifying Structural and Non‐structural Expectations in Relative Clause Processing

Chen

Hale

2021

Cognitive Science

View full text Add to dashboard Cite

Information‐theoretic complexity metrics, such as Surprisal (Hale, 2001; Levy, 2008) and Entropy Reduction (Hale, 2003), are linking hypotheses that bridge theorized expectations about sentences and observed processing difficulty in comprehension. These expectations can be viewed as syntactic derivations constrained by a grammar. However, this expectation‐based view is not limited to syntactic information alone. The present study combines structural and non‐structural information in unified models of word‐by‐word sentence processing difficulty. Using probabilistic minimalist grammars (Stabler, 1997), we extend expectation‐based models to include frequency information about noun phrase animacy. Entropy reductions derived from these grammars faithfully reflect the asymmetry between subject and object relatives (Staub, 2010; Staub, Dillon, & Clifton, 2017), as well as the effect of animacy on the measured difficulty profile (Lowder & Gordon, 2012; Traxler, Morris, & Seely, 2002). Visualizing probability distributions on the remaining alternatives at particular parser states allows us to explore new, linguistically plausible interpretations for the observed processing asymmetries, including the way that expectations about the relativized argument influence the processing of particular types of relative clauses (Wagers & Pendleton, 2016).

show abstract

Lossy‐Context Surprisal: An Information‐Theoretic Model of Memory Effects in Sentence Processing

Cited by 122 publications

References 127 publications

Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty

Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty

Word Order Typology Interacts With Linguistic Complexity: A Cross‐Linguistic Corpus Study

Quantifying Structural and Non‐structural Expectations in Relative Clause Processing

Contact Info

Product

Resources

About