Parietal cortex integrates contextual and saliency signals during the encoding of natural scenes in working memory

Abstract

The Brief presentation of a complex scene entails that only a few objects can be selected, processed indepth, and stored in memory. Both low-level sensory salience and high-level context-related factors (e.g., the conceptual match/mismatch between objects and scene context) contribute to this selection process, but how the interplay between these factors affects memory encoding is largely unexplored. Here, during fMRI we presented participants with pictures of everyday scenes. After a short retention interval, participants judged the position of a target object extracted from the initial scene. The target object could be either congruent or incongruent with the context of the scene, and could be located in a region of the image with maximal or minimal salience. Behaviourally, we found a reduced impact of saliency on visuospatial working memory performance when the target was out-of-context. Encoding-related fMRI results showed that context-congruent targets activated dorsoparietal regions, while context-incongruent targets de-activated the ventroparietal cortex. Saliency modulated activity both in dorsal and ventral regions, with larger context-related effects for salient targets. These findings demonstrate the joint contribution of knowledge-based and saliency-driven attention for memory encoding, highlighting a dissociation between dorsal and ventral parietal regions.

The COGs (context, object, and goals) in multisensory processing

“…As already discussed in Section 1.1, observers'tolerance to interstimulus delays during judgements of simultaneity of AV stimuli is modulated by the particular demands of thesimultaneity-judgement task (Stevenson and Wallace 2013; but this tolerance likewise depends on the stimulus category, see below). Collectively, these findings demonstrate that the role of matching multisensory stimulus features can be better understood in situations where the congruence of object features is task-irrelevant (Mastroberardino et al 2015;Santangelo et al 2015).…”

“…Iordanescu et al 2009;van Ee et al 2009;Orchard-Mills et al 2013a, 2013bNardo et al 2014;Mastroberardino et al 2015).While these findings are in line with the influence of feature-based unisensory attention (Desimone and Duncan 1995), whether it applies to multisensory situations remains unclear. To investigate this possibility, Matusz and Eimer (2013) employed multi-stimulus visual displays and instructed participants to search for targets defined by a visual feature alone (e.g.…”

Changing What You See by Changing What You Know: The Role of Attention

“…Perceiving the image as meaningful helps people perform a simple perceptual task—determining whether two Mooney images are identical or not. These behavioral improvements were related to differences in early visual processing (specifically, larger amplitudes of the P1 EEG signal, Samaha et al, 2016; see also Abdel Rahman and Sommer, 2008). Contra Pylyshyn’s (1999, p. 357) statement that “verbal hints [have] little effect on recognizing fragmented figures”, we find that not only do verbal hints greatly enhance recognition, but they facilitate visual discrimination.…”

“…In case of the Mooney image depicted in the last row of Figure 3 , even superordinate linguistic cues like “animal” and “musical instrument” aid in recognition of the images. More specific cues (e.g., the word “trumpet”) are predictably more effective (Samaha et al, 2016). In other work, we have shown using hearing a verbal cue affects visual processing within 100 ms. of visual onset (Boutonnet and Lupyan, 2015), results that we interpret as showing that verbal cues activate visual representations, establishing “priors” that change how subsequent stimuli are processed (Edmiston and Lupyan, 2015, 2017; Lupyan and Clark, 2015).…”

Preferential Processing of Social Features and Their Interplay with Physical Saliency in Complex Naturalistic Scenes

“…Following the seminal work of Itti and Koch, a diverse set of computational saliency approaches has been developed (for reviews, see e.g., Judd et al, 2012; Borji and Itti, 2013). Although research provided empirical support for saliency-based attention models (e.g., when humans freely viewed or memorized visual scenes; Parkhurst et al, 2002; Foulsham and Underwood, 2008), there are several recent studies indicating circumstances under which these models work less well or fail completely (e.g., under presence of top–down influences from visual search tasks; Foulsham and Underwood, 2007; Henderson et al, 2007; Einhäuser et al, 2008a; see also e.g., Stirk and Underwood, 2007; Einhäuser et al, 2008b; Santangelo, 2015; Santangelo et al, 2015; but see e.g., Borji et al, 2013; Spotorno et al, 2013). Importantly, prior research also shed light on the power of saliency-based predictions in the context of social information.…”