Fixation-related brain activity during semantic integration of object-scene information

Abstract

In vision science, a topic of great interest and considerable controversy is the processing of objects that are (in)consistent with the overall meaning of the scene in which they occur. How quickly can we access the semantic properties of objects and does this happen before the object is directly looked at? Here we brought novel evidences to this debate by co-registering eye-movements and EEG while participants freely explored photographs of indoor scenes. Each scene contained a target object that was either consistent or inconsistent with the scene context (e.g., toothpaste vs. flashlight in a bathroom). Eye-movement behaviour showed that inconsistent objects were not only more effortful to process (i.e., looked at longer), but also prioritized over consistent objects while they were still in extrafoveal vision (i.e., looked at earlier after scene onset). In fixation-related brain potentials (FRPs), we used a linear deconvolution technique to compare the activity elicited by semantically consistent versus inconsistent objects at scene onset (i.e. in the stimulus-locked ERP), during the target fixation (t), and during the preceding fixation (t-1). We demonstrate a fixation-related N300/N400 effect, whereby inconsistent objects elicited a larger negative shift of FRP activity than consistent objects, already at the pre-target fixation t-1 and throughout fixation t. Taken together, our results suggest that object meaning can be, at least partly, extracted in extrafoveal vision, and highlight how attentional and neural mechanisms can be simultaneously investigated to uncover the mechanisms of object-scene integration during natural viewing.

Extra-foveal Processing of Object Semantics Guides Early Overt Attention During Visual Search

“…Moores et al (2003) and Belke et al (2008) had also reported semantic relatedness effects on initial eye movements during search. However, these earlier studies have been criticised by Daffron and Davis (2016) who suggested that effects of semantic relatedness might have been confounded by the repeated presentation of the visual stimuli to the participants. Thus, overt attention might have been biased by remembered visual features of the objects, rather than by their semantic features.…”

“…Moores et al (2003) and Belke et al (2008) reported semantic relatedness effects on the very first saccadic eye movement after the onset of the object array. However, Daffron and Davis (2016) claimed that this evidence might have been confounded by the repeated exposure of the stimuli to the participants. For example, in Belke et al(2008), participants inspected the visual stimuli (line drawings of objects) before the experiment began, thus raising the concern that eye movements were guided by the memory of the visual features of the stimuli rather than by their semantics.…”

The intersection between the oculomotor and hippocampal memory systems: empirical developments and clinical implications

“…That is, greater viewing is directed at the areas of interest within the environment that are not well represented within, or violate the expectations from, a viewer's schemas, and this occurs for the purpose of continually forming new memories and updating knowledge structures. This notion is supported by combined eye movement–ERP findings in which longer gaze durations were directed to, and larger neural signatures indicative of semantic processing preceded and followed the initial fixation on, a target object that was inconsistent with the meaning of the scene, compared with a consistent target object …”

“…41 That is, greater viewing is directed at the areas of interest within the environment that are not well represented within, or violate the expectations from, a viewer's schemas, and this occurs for the purpose of continually forming new memories and updating knowledge structures. This notion is supported by combined eye movement-ERP findings in which longer gaze durations were directed to, and larger neural signatures indicative of semantic processing preceded and followed the initial fixation on, a target object that was inconsistent with the meaning 161 There is also a reasonable consensus that the pattern of visual exploration is, at the very least, influenced by existing memories. 2 Whether eye movements have a functional role at retrieval, or the influence of memory on the pattern of visual exploration at retrieval is merely epiphenomenal, is an open question.…”

Peer Review #2 of "Unfold: an integrated toolbox for overlap correction, non-linear modeling, and regression-based EEG analysis (v0.2)"

“…However, while RIDE has been successfully used to separate stimulus-and response-related ERP components (Ouyang et al, 2011(Ouyang et al, , 2015; it does not support continuous predictors and is intended for a small number of overlapping events. In recent years, an alternative deconvolution method based on the linear model has been proposed and successfully applied to the overlap problem (Lütkenhöner, 2010;Litvak, Jha, Flandin, & Friston, 2013;Spitzer et al, 2016;Kristensen, Guerin-Dugué, & Rivet, 2017;Sassenhagen, 2018;Cornelissen, Sassenhagen, & Võ, 2019;Coco, Nuthmann, & Dimigen, 2018;Bigdely-Shamlo et al, 2018). This deconvolution approach was first applied extensively to fMRI data (Dale & Buckner, 1997) where the slowly varying BOLD signal overlaps between subsequent events.…”