Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions

2019

J Cult Cogn Sci

Self Cite

Predicting variability in context effects is a timely enterprise considering that psycho-and neurolinguistic research has assessed how language processing depends on the perceived context, the body, and long-term linguistic knowledge of the language user. The current evidence suggests that some context effects may be systematically more robust than others and that language user characteristics are an influential modulator of context-sensitive comprehension. Reviewing psycholinguistic evidence, I argue for constrained contextual variability. Variability in context effects is predicted by characteristics of the language user and world-language relations. But extant findings also suggest generalizability beyond such variation, thus imposing constraint on theoretical prediction of context effects via relative (not absolute) processing preferences. Keywords Visually-situated language comprehension Á Predicting context effects Á Variability Á Systematicity Á Language user characteristics A focus on language and linguistic experience in theories of language comprehension

Section: Variation In Expectation-based Comprehension and In Context mentioning

confidence: 99%

Section: Variation In Expectation-based Comprehension and In Context mentioning

confidence: 99%

Predicting (variability of) context effects in language comprehension

2019

J Cult Cogn Sci

Self Cite

“…Alternatively, do they modulate comprehension in distinct ways (Knoeferle et al, 2014;Kreysa et al, 2018;Knoeferle, 2019)-much like different sorts of linguistic knowledge are processed distinctly (Hagoort, 2003;Huettig and McQueen, 2007;Bastiaansen and Hagoort, 2015;Lapinskaya et al, 2016)? Initial evidence suggests that referential and non-referential world-language relations modulate comprehension and visual attention in distinct ways with referential relations eliciting more visual attention to objects than non-referential relations (Cooper, 1974).…”

Section: Introductionmentioning

confidence: 99%

“…A range of studies on distinct world-language relations (Knoeferle et al, 2014;Kreysa et al, 2018;Kröger et al, 2018) pitted action verbs and their referents against other cues and world-language relations (speaker facial emotion, role relations, speaker gaze, or a wiggle motion of a character). Unlike the case of an action verb denoting a depicted action, these other worldlanguage relations were non-referential.…”

Section: Introductionmentioning

confidence: 99%

Referential vs. Non-referential World-Language Relations: How Do They Modulate Language Comprehension in 4 to 5-Year-Olds, Younger, and Older Adults?

Maquate

2021

Front. Psychol.

Self Cite

Age has been shown to influence language comprehension, with delays, for instance, in older adults' expectations about upcoming information. We examined to what extent expectations about upcoming event information (who-does-what-to-whom) change across the lifespan (in 4- to 5-year-old children, younger, and older adults) and as a function of different world-language relations. In a visual-world paradigm, participants in all three age groups inspected a speaker whose facial expression was either smiling or sad. Next they inspected two clipart agents (e.g., a smiling cat and a grumpy rat) depicted as acting upon a patient (e.g., a ladybug tickled by the cat and arrested by the rat). Control scenes featured the same three characters without the action depictions. While inspecting the depictions, comprehenders listened to a German sentence [e.g., Den Marienkäfer kitzelt vergnügt der Kater; literally: “The ladybug (object/patient) tickles happily the cat (subject/agent)”]. Referential verb-action relations (i.e., when the actions were present) could, in principle, cue the cat-agent and so could non-referential relations via links from the speaker's smile to “happily” and the cat's smile. We examined variation in participants' visual anticipation of the agent (the cat) before it was mentioned depending on (a) participant age and (b) whether the referentially mediated action depiction or the non-referentially associated speaker smile cued the agent. The action depictions rapidly boosted participants' visual anticipation of the agent, facilitating thematic role assignment in all age groups. By contrast, effects of the non-referentially cued speaker smile emerged in the younger adults only. We outline implications of these findings for processing accounts of the temporally coordinated interplay between listeners' age-dependent language comprehension, their interrogation of the visual context, and visual context influences.

“…Crucially, non-linguistic information can also facilitate real-time language processing of canonical and non-canonical grammatical sentences (e.g., Knoeferle et al, 2005 ; Carminati and Knoeferle, 2013 ). Social cues, such as, for example, a speaker's emotional facial expression (Carminati and Knoeferle, 2013 ), a speaker's gaze shift (Kreysa et al, 2018 ), or the speaker's voice information (Van Berkum et al, 2008 ), can elicit expectations on the part of the listener (just like other non-linguistic cues), and these can in turn influence the processing of upcoming linguistic information. However, existing research has focused mostly on assessing how object- and action-related visual information influences spoken language comprehension.…”

Section: Introductionmentioning

confidence: 99%

Integration of Social Context vs. Linguistic Reference During Situated Language Processing

Maquate

2021

Front. Psychol.

Self Cite

Research findings on language comprehension suggest that many kinds of non-linguistic cues can rapidly affect language processing. Extant processing accounts of situated language comprehension model these rapid effects and are only beginning to accommodate the role of non-linguistic emotional, cues. To begin with a detailed characterization of distinct cues and their relative effects, three visual-world eye-tracking experiments assessed the relative importance of two cue types (action depictions vs. emotional facial expressions) as well as the effects of the degree of naturalness of social (facial) cues (smileys vs. natural faces). We predicted to replicate previously reported rapid effects of referentially mediated actions. In addition, we assessed distinct world-language relations. If how a cue is conveyed matters for its effect, then a verb referencing an action depiction should elicit a stronger immediate effect on visual attention and language comprehension than a speaker's emotional facial expression. The latter is mediated non-referentially via the emotional connotations of an adverb. The results replicated a pronounced facilitatory effect of action depiction (relative to no action depiction). By contrast, the facilitatory effect of a preceding speaker's emotional face was less pronounced. How the facial emotion was rendered mattered in that the emotional face effect was present with natural faces (Experiment 2) but not with smileys (Experiment 1). Experiment 3 suggests that contrast, i.e., strongly opposing emotional valence information vs. non-opposing valence information, might matter for the directionality of this effect. These results are the first step toward a more principled account of how distinct visual (social) cues modulate language processing, whereby the visual cues that are referenced by language (the depicted action), copresent (the depicted action), and more natural (the natural emotional prime face) tend to exert more pronounced effects.