Investigating the role of entropy in sentence processing

Linzen, Tal; Jaeger, T. Florian

doi:10.3115/v1/w14-2002

Cited by 14 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Surprisal is often used to quantify the processing cost of linguistic units, as a reliable correlation has been established in that a more surprising and more informative word or utterance takes more time to read and/or more effort to process (Demberg and Keller, 2008; Smith and Levy, 2013). A similar connection has been found between effort and the concept of entropy reduction, according to which a more informative word reduces uncertainty about the entire sentence structure more sand hence might take longer to read (Linzen and Jaeger, 2014). This concept may also account for recent findings by Maess et al (2016).…”

Section: Surprisal and Entropy Reduction In The Vwpsupporting

confidence: 56%

The Influence of Visual Uncertainty on Word Surprisal and Processing Effort

2018

View full text Add to dashboard Cite

A word’s predictability or surprisal, as determined by cloze probabilities or language models (Frank, 2013) is related to processing effort, in that less expected words take more effort to process (Hale, 2001; Lau et al., 2013). A word’s surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999). How visual context affects surprisal and processing effort, however, remains unclear. Here, we present a series of four studies providing evidence on how visually-determined probabilistic expectations for a spoken target word, as indicated by anticipatory eye movements, predict graded processing effort for that word, as assessed by a pupillometric measure (the Index of Cognitive Activity, ICA). These findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.

show abstract

Section: Surprisal and Entropy Reduction In The Vwpsupporting

confidence: 56%

The Influence of Visual Uncertainty on Word Surprisal and Processing Effort

2018

View full text Add to dashboard Cite

show abstract

“…If the P600 indeed reflects syntactic reanalysis, we could therefore have seen surprisal effects on the P600. Even an entropy-reduction effect could not have been excluded in advance, considering that Hale (2003) and Linzen and Jaeger (2014) demonstrate that some garden paths can be viewed as effects of entropy reduction rather then surprisal. However, the P600 has also been found in cases that do not involve increased syntactic processing difficulty (e.g., Hoeks, Stowe, & Doedens, 2004;Kuperberg, Kreher, Sitnikova, Caplan, & Holcomb, 2007;Regel, Gunter, & Friederici, 2011;Van Berkum, Koornneef, Otten, & Nieuwland, 2007).…”

Section: Late Positivitiesmentioning

confidence: 93%

The ERP response to the amount of information conveyed by words in sentences

et al. 2015

View full text Add to dashboard Cite

Reading times on words in a sentence depend on the amount of information the words convey, which can be estimated by probabilistic language models. We investigate whether event-related potentials (ERPs), too, are predicted by information measures. Three types of language models estimated four different information measures on each word of a sample of English sentences. Six different ERP deflections were extracted from the EEG signal of participants reading the same sentences. A comparison between the information measures and ERPs revealed a reliable correlation between N400 amplitude and word surprisal. Language models that make no use of syntactic structure fitted the data better than did a phrase-structure grammar, which did not account for unique variance in N400 amplitude. These findings suggest that different information measures quantify cognitively different processes and that readers do not make use of a sentence's hierarchical structure for generating expectations about the upcoming word.

show abstract

“…This study also investigates the extent to which non-DLT measures of processing complexity can predict syntactic choice. While DLT predicts an influence from the length of dependencies, increased memory load may also reduce 3 For a discussion of embedding depth as part of the prediction process, please refer to Linzen and Jaeger (2014;. the amount of resources available to process language (Chomsky & Miller 1963, Schuler, AbdelRahman, Miller, & Schwartz 2010, Yngve 1960.…”

Section: Introductionmentioning

confidence: 99%

Investigating locality effects and surprisal in written English syntactic choice phenomena

et al. 2016

View full text Add to dashboard Cite

We investigate the extent to which syntactic choice in written English is influenced by processing considerations as predicted by Gibson's (2000) Dependency Locality Theory (DLT) and Surprisal Theory (Hale, 2001; Levy, 2008). A long line of previous work attests that languages display a tendency for shorter dependencies, and in a previous corpus study, Temperley (2007) provided evidence that this tendency exerts a strong influence on constituent ordering choices. However, Temperley's study included no frequency-based controls, and subsequent work on sentence comprehension with broad-coverage eye-tracking corpora found weak or negative effects of DLT-based measures when frequency effects were statistically controlled for (Demberg & Keller, 2008; van Schijndel, Nguyen, & Schuler 2013; van Schijndel & Schuler, 2013), calling into question the actual impact of dependency locality on syntactic choice phenomena. Going beyond Temperley's work, we show that DLT integration costs are indeed a significant predictor of syntactic choice in written English even in the presence of competing frequency-based and cognitively motivated control factors, including n-gram probability and PCFG surprisal as well as embedding depth (Wu, Bachrach, Cardenas, & Schuler, 2010; Yngve, 1960). Our study also shows that the predictions of dependency length and surprisal are only moderately correlated, a finding which mirrors Dember & Keller's (2008) results for sentence comprehension. Further, we demonstrate that the efficacy of dependency length in predicting the corpus choice increases with increasing head-dependent distances. At the same time, we find that the tendency towards dependency locality is not always observed, and with pre-verbal adjuncts in particular, non-locality cases are found more often than not. In contrast, surprisal is effective in these cases, and the embedding depth measures further increase prediction accuracy. We discuss the implications of our findings for theories of language comprehension and production, and conclude with a discussion of questions our work raises for future research.

show abstract

Investigating the role of entropy in sentence processing

Cited by 14 publications

References 21 publications

The Influence of Visual Uncertainty on Word Surprisal and Processing Effort

The Influence of Visual Uncertainty on Word Surprisal and Processing Effort

The ERP response to the amount of information conveyed by words in sentences

Investigating locality effects and surprisal in written English syntactic choice phenomena

Contact Info

Product

Resources

About