23 Neurocognitive Mechanisms of Human Comprehension

Abstract

This chapter begins with a discussion of evidence for distinctions between two semantic comprehension systems in the language domain: a system that maps the perceived information on graded semantic representations and a system that utilizes particular semantic requirements of verbs. It then reviews similar research using static and motion pictures. It argues that the two mechanisms of language comprehension might be analogous to the systems that use graded semantic representations and action-based requirements to make sense of the visual world. The experiments that are reviewed in this chapter examine questions of both how comprehenders understand relationships between the elements within individual events and how they understand the relationships between events. Experiments that have used event-related potentials (ERPs) are also highlighted.

The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension

“…a man attempting to cut bread with an iron following a context showing him in the kitchen with the bread on a bread board (Sitnikova et al, 2008b; Sitnikova et al, 2003). We suggested that the P600 was triggered by participants’ detection of this strong event structure prediction violation, and that it reflected prolonged attempts to restructure and make sense of the input (Kuperberg, 2007; Sitnikova et al, 2008a, 2008b; see also Võ & Wolfe, 2013). In the present study, we offer a similar interpretation.…”

“…This of course does not imply that the structures that we draw upon to make sense of language, music, and visual narrative are the same; each of these cognitive domains differ both in their basic units (words, notes, images) and in the rules by which they are combined (syntax, harmony, narrative). What it does suggest, however, is that constituent structure is not specific to language, and that the brain draws upon similar neurocognitive mechanisms or common computational principles to analyze structure across multiple domains (Corballis, 1991; Hoen & Dominey, 2000; Jackendoff, 2011; Patel, 2003; Sitnikova et al, 2008a; Võ & Wolfe, 2013). …”

“…The P600 is most likely to be triggered when an input violates a strong, high certainty single structural expectation established by a context, particularly when this context is also semantically constraining (Kuperberg, 2007, 2013). It is believed to reflect prolonged attempts to make sense of the input (Kuperberg, 2007, 2013; Sitnikova, Holcomb, & Kuperberg, 2008a). Notably, both the left anterior negativity and P600 effects are distinct from the well-known N400 effect—a widespread negativity between 300-500ms that is modulated by both words (Kutas & Hillyard, 1980) and images (Barrett & Rugg, 1990; Barrett, Rugg, & Perrett, 1988) that match versus mismatch contextual expectations about the semantic features of upcoming input, rather than expectations about its grammatical structure (Kutas & Federmeier, 2011).…”

The notion of the motion: The neurocognition of motion lines in visual narratives

“…Because P600 effects appear across domains, several researchers have posited that it reflects a domain-general process of attempting to integrate an input into a structure in the face of a prediction error, possibly causing an update or revision of the wider representation of context (Christiansen, Conway, & Onnis, 2011; Cohn, Jackendoff, Holcomb, & Kuperberg, 2014; Kuperberg, 2013; Sitnikova, Holcomb, & Kuperberg, 2008a; Võ & Wolfe, 2013). This general “prediction error” may also be connected to a more frontally distributed positivity appearing after 500ms from the stimulus onset (Kuperberg, 2013; Van Petten & Luka, 2012).…”

What do we mean by prediction in language comprehension?

“…This knowledge might also include the necessary and likely temporal, spatial, and causal relationships that link multiple events and states together to form sequences of events. The latter are sometimes referred to as scripts, frames, or narrative schemas (Fillmore, 2006; Schank & Abelson, 1977; Sitnikova, Holcomb, & Kuperberg, 2008; Wood & Grafman, 2003; Zwaan & Radvansky, 1998). …”