Different mechanisms for role relations versus verb–action congruence effects: Evidence from ERPs in picture–sentence verification

Abstract

Extant accounts of visually situated language processing do make general predictions about visual context effects on incremental sentence comprehension; these, however, are not sufficiently detailed to accommodate potentially different visual context effects (such as a scene-sentence mismatch based on actions versus thematic role relations, e.g., Altmann & Kamide, 2007;Knoeferle & Crocker, 2007;Taylor & Zwaan, 2008;Zwaan & Radvansky, 1998).To provide additional data for theory testing and development, we collected event-related brain potentials (ERPs) as participants read a subject-verbobject sentence (500 ms SOA in Experiment 1 and 300 ms SOA in Experiment 2), and post-sentence verification times indicating whether or not the 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 verb and/or the thematic role relations matched a preceding picture (depicting two participants engaged in an action). Though incrementally processed, these two types of mismatch yielded different ERP effects. Role-relation mismatch effects emerged at the subject noun as anterior negativities to the mismatching noun, preceding action mismatch effects manifest as centro-parietal N400s greater to the mismatching verb, regardless of SOAs. These two types of mismatch manipulations also yielded different effects post-verbally, correlated differently with a participant's mean accuracy, verbal working memory and visual-spatial scores, and differed in their interactions with SOA. Taken together these results clearly implicate more than a single mismatch mechanism for extant accounts of picture-sentence processing to accommodate.

Predicting (variability of) context effects in language comprehension

“…Robust referential and action effects: a processing preference? This variability notwithstanding, action effects, for example, have been robustly observed in numerous studies, using both eye-tracking and eventrelated brain potentials as measures; the effects emerged as soon as the verb mediated an action Crocker 2006, 2007;Knoeferle et al 2008Knoeferle et al , 2011Knoeferle et al , 2014. For instance, when comprehenders encountered a verb (e.g., 'paints') in the structurally ambiguous spoken sentence 'The princess (amb.…”

“…On the basis of these and other results, it has been argued (Knoeferle et al 2011) that verification of language against context seems to be a fundamental part of language processing in context. However, despite its occasional use in language comprehension research (Goolkasian 1996;Singer 2006;Underwood et al 2004) insights from the verification task have had minimal impact on psycholinguistic theories of online sentence comprehension, but see Knoeferle et al (2014).…”

“…But each of the context-based strands of research from the 70s and 80-verification, semantic activation, the inferential and script-based construction of mental models of the world, and real-time interaction of language comprehension with visual attention-has influenced recent psycholinguistic modeling research. Verification mechanisms have been integrated in sentence processing accounts (Knoeferle et al 2014;Van Herten et al 2006). Priming research has made its way into psycholinguistic theory in the form of 'alignment' between interlocutors (viz.…”

“…• Building structure and assigning an interpretation (by interpretation we mean a mental representation of sentence meaning, informed by linguistic, and world knowledge, and the immediate linguistic and non-linguistic context) • Grounding the interpretation in a model of the world/visual context via (internal/visual) attention (by 'model' we mean mental representations of the immediate non-linguistic context, (see Knoeferle and Crocker 2007;Knoeferle et al 2014)) • Verifying the structure and interpretation against representations of the world/visual context and revising as necessary…”

Effects of Emotional State on Processing Conceptual Mismatches: ERPs in a Picture-Sentence Matching Task

“…This, in combination with the abstract scenarios depicted in the present study (e.g., The star stands behind the circle), may explain the absence of an interaction between language and emotion for these specific pictorial mismatches. To address this issue, future studies should examine the effects of emotional state on the processing of conceptual mismatches using animate (hu-Journal of Behavioral and Brain Science man-like) creatures involved in more natural actions (see for example for the use of more natural materials [12] [35]). …”