Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Abstract

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to comprehension informed by comprehenders' experience. Among the examined aspects of the visual context are actions, events, a speaker's gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eyetracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension. Video Link The video component of this article can be found at https://www.jove.com/video/57694/ 2). In this 'visual world' eye-tracking version, the inspection of objects is guided by language. When the comprehenders hear the zebra, for instance, their inspection of a zebra on the screen is taken to reflect that they are thinking about the animal. In what is known as the visual world paradigm, a comprehender's eye gaze is taken to reflect spoken language comprehension and the activation of associated knowledge (e.g., listeners also inspect the zebra when they hear grazing, indicating an action performed by the zebra) 2. Such inspections suggest a systematic link between language-world relations and eye movements 2. A common way to quantify this link is by computing the proportion of looks to different predetermined regions on a screen. This allows researchers to directly compare (across conditions, by participants and items) the amount of attention given to different objects at a particular time and how these values change in millisecond resolution.

Common Ground Information Affects Reference Resolution: Evidence From Behavioral Data, ERPs, and Eye-Tracking

“…In addition, EEG allows to draw inferences about the underlying neural mechanisms of CG integration and can be directly compared to the findings of Sikos et al (2019). Our second study (eye-tracking, Experiment 2) with a mostly identical design to the ERP-Experiment allows for a descriptive alignment of eye-tracking results with our ERP data and provides further insights into the interaction of language comprehension and the perception of the visual world (for a recent short methodological overview see Rodriguez Ronderos et al, 2018). In addition to the ERP analysis, we appended an exploratory time-frequency analysis (TFA) of the EEG data in the Supplementary Material, which might provide insights about the mechanisms underlying CG processing.…”

Syntactic and Semantic Influences on the Time Course of Relative Clause Processing: The Role of Language Dominance

“…In the present study, we used the same experimental paradigm as the two studies described above but treated both language dominance and time as continuous variables in order to gain a finer-grained understanding of the relationship between individual-level language dominance and the time course of processing. An important benefit of continuous monitoring (vs. comprehension questions, or aggregated time bins) is that the gaze record provides insights into how comprehension processes unfold over time on a scale of milliseconds and how the immediate visual context contributes to spoken sentence comprehension [39]. In particular, we examined fixations before the onset of the relative clause in order to examine processing preferences in the absence of disambiguating linguistic information, i.e., predictive processing.…”