Semantic Entropy in Language Comprehension

Abstract

Language is processed on a more or less word-by-word basis, and the processing difficulty induced by each word is affected by our prior linguistic experience as well as our general knowledge about the world. Surprisal and entropy reduction have been independently proposed as linking theories between word processing difficulty and probabilistic language models. Extant models, however, are typically limited to capturing linguistic experience and hence cannot account for the influence of world knowledge. A recent comprehension model by Venhuizen, Crocker, and Brouwer (2019, Discourse Processes) improves upon this situation by instantiating a comprehension-centric metric of surprisal that integrates linguistic experience and world knowledge at the level of interpretation and combines them in determining online expectations. Here, we extend this work by deriving a comprehension-centric metric of entropy reduction from this model. In contrast to previous work, which has found that surprisal and entropy reduction are not easily dissociated, we do find a clear dissociation in our model. While both surprisal and entropy reduction derive from the same cognitive process—the word-by-word updating of the unfolding interpretation—they reflect different aspects of this process: state-by-state expectation (surprisal) versus end-state confirmation (entropy reduction).

Neurobehavioral Correlates of Surprisal in Language Comprehension: A Neurocomputational Model

“…That is, the more likely the interpretation at t given the interpretation at t − 1, the lower the Surprisal induced by word w t (see Venhuizen et al, 2019b , for a similar DSS-derived conceptualization of Entropy).…”

Quantifying Structural and Non‐structural Expectations in Relative Clause Processing

“…Recent work has also started to integrate world knowledge and linguistic experience in models of online sentence comprehension, including both surprisal (Venhuizen, Crocker, & Brouwer, 2019a) and entropy reduction (Venhuizen, Crocker, & Brouwer, 2019b). The modeling framework described in this paper is flexible enough to include discourse information as an ingredient, for example, by quantifying the expectation of ongoing discourse referents as a degree of uncertainty.…”

Less is More/More Diverse: On The Communicative Utility of Linguistic Conventionalization

“…As a formal index of paradigmatic variability we use entropy, which measures the contribution of linguistic units (e.g., words) in predicting linguistic choice in bits of information. Using entropy provides us with a link to a communicative interpretation, as it is a well-established measure of communicative efficiency with implications for cognitive processing (Linzen and Jaeger, 2016;Venhuizen et al, 2019); also, entropy is negatively correlated with distance in (word embedding) spaces which in turn shows cognitive reflexes in certain language processing tasks (Mitchel et al, 2008;Auguste et al, 2017). In terms of domain we focus on science, looking at the diachronic development of scientific English from the 17th century to modern time.…”