Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

Abstract

The present chapter reviews the literature on visually situated language comprehension against the background that most theories of real-time sentence comprehension have ignored rich non-linguistic contexts. However, listeners' eye movements to objects during spoken language comprehension, as well as their eventrelated brain potentials (ERPs) have revealed that non-linguistic cues play an important role for real-time comprehension. In fact, referential processes are rapid and central in visually situated spoken language comprehension and even abstract words are rapidly grounded in objects through semantic associations. Similar ERP responses for non-linguistic and linguistic effects on comprehension suggest these two information sources are on a par in informing language comprehension. ERPs further revealed that non-linguistic cues affect lexical-semantic as well as compositional processes, thus further cementing the role of rich non-linguistic context in language comprehension. However, there is also considerable ambiguity in the linking between comprehension processes and each of these two measures (eye movements and ERPs). Combining eye-tracking and event-related brain potentials would improve the interpretation of individual measures and thus insights into visually-situated language comprehension.Knoeferle, P. (to appear). Language comprehension in rich non-linguistic contexts: combining eye tracking and event-related brain potentials. In: Roel, Williams (Ed.). Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press.

Bilingual switching between languages and listeners: Insights from immersive virtual reality

“…As a next step forward towards experimentally mimicking naturally occurring instances of bilingual language switching, future studies could replace the picture-naming component used in language production paradigms by a situation in which bilinguals refer to various 3D objects in realistic 3D environments for speakers with different language backgrounds. Such step-by-step improvements of existing paradigms will help moving towards neurocognitive theories of language that do justice to the multimodal richness and dynamics of naturally occurring human communication (Hari et al, 2015;Knoeferle, 2015). Moreover, the advantage of a step-by-step approach away from an existing paradigm is that it allows one to exactly pinpoint which factor drives a potential deviation in result patterns compared to results obtained in a traditional experimental paradigm.…”

Visually Situated Language Comprehension

“…From a theoretical viewpoint, we will want to complement a probabilistic level of analysis with an explanatory psycholinguistic account (detailing why comprehenders prioritize referential relations). For instance, we might reason that this behavior in young adults has a developmental basis and that the importance of referential relations for word learning in infancy is at the origin of the referential priority in young adults' language comprehension (Knoeferle 2015b). An alternative (or complementary) explanation is epistemic in nature.…”

“…We can also assess the importance of referential world-language relations by examining children's language learning and comprehension. If we assume continuity of language comprehension from early childhood into adulthood, then a central role of visual context in adult Visually Situated Language Comprehension 75 comprehension would predict clear referential effects at developmental stages (see Knoeferle 2015b). It's not implausible that children have a preference for establishing reference since even young children rapidly fixate the picture of a word they have recognized (see Fernald, McRoberts, and Swingley 2001;Hollich, Hirsh-Pasek, and Golinkoff 2000;Swingley, Pinto, and Fernald 1999).…”

“…To gain insight into potential systematicity, extant research has asked whether different cues or world-language relations are on a par in informing language processing. Visually situated approaches have for instance argued that all else being equal, comprehenders exhibit a 'referential priority / preference' (see Knoeferle andCrocker 2006, 2007;Knoeferle, Carminati, Abashidze, and Essig 2011). By 'referential', we mean the relation between a word and the object it denotes, and this includes at least noun-object and verb-action relations ( Jackendoff 2002).…”

Motor resonance during linguistic processing as shown by EEG in a naturalistic VR environment

“…This very decontextualization could in turn affect how language is processed. Indeed, real-world language processing generally occurs in much richer environments and, importantly, interlocutors, social context and physical cues have a strong influence on how language is understood (Knoeferle, 2015). In the last two decades, the discrepancy between real-life language processing and that which takes place in an experimental environment has been brought to light in the context of embodied cognition (Tromp, Peeters, Meyer, & Hagoort, 2018), according to which our bodily states and actions are heavily implicated in how we communicate and process information (Atkinson, 2010).…”