Meaning Guides Attention during Real-World Scene Description

Abstract

Intelligent analysis of a visual scene requires that important regions be prioritized and attentionally selected for preferential processing. What is the basis for this selection? Here we compared the influence of meaning and image salience on attentional guidance in real-world scenes during two free-viewing scene description tasks. Meaning was represented by meaning maps capturing the spatial distribution of semantic features. Image salience was represented by saliency maps capturing the spatial distribution of image features. Both types of maps were coded in a format that could be directly compared to maps of the spatial distribution of attention derived from viewers’ eye fixations in the scene description tasks. The results showed that both meaning and salience predicted the spatial distribution of attention in these tasks, but that when the correlation between meaning and salience was statistically controlled, only meaning accounted for unique variance in attention. The results support theories in which cognitive relevance plays the dominant functional role in controlling human attentional guidance in scenes. The results also have practical implications for current artificial intelligence approaches to labeling real-world images.

Where the action could be: Speakers look at graspable objects and meaningful scene regions when describing potential actions.

“…For those 15 scenes, the authors confirmed that objects in the foreground of the scene were within reach of the scene's viewpoint. The remaining scenes were drawn from other studies: 4 from Xu et al (2014) and 1 from Cullimore, Rehrig, Henderson, and Ferreira (2018). Three scenes depicted outdoor environments (two beach scenes), and 17 depicted indoor environments (eight kitchens, five dining areas).…”

“…These assumptions follow from saliencybased theories of visual attention (e.g., Itti & Koch, 2001;Wolfe & Horowitz, 2017) in which visual attention is drawn to peaks in image salience, defined as regions that stand out from the rest on the basis of low-level features (e.g., luminance, orientation, color). An alternative explanation for what guides attention in scenes comes from cognitive guidance theory (Henderson, 2007): The cognitive system directs gaze to informative areas for detailed scrutiny (Henderson, Malcolm, & Schandl, 2009;Henderson, 2017;Henderson & Hayes, 2017;Henderson, Hayes, Rehrig, & Ferreira, 2018). Image salience plays a lesser role in cognitive guidance theory, initially helping to determine the lay of the land, but scene regions are selected for attention on the basis of meaning, modulated by the viewer's current goals (both meaning and task-relevance guide attention).…”

Initiation of utterance planning in response to pre-recorded and “live” utterances

“…The analysis of the participants' eye movements showed that during their own turn, they almost exclusively looked at their own objects, regardless of whether or not they were tapping and regardless of the presence or absence of the confederate. This finding adds to a large body of findings from multiple-object naming studies demonstrating that speakers typically look at the objects they name or describe, most likely because this facilitates the recognition of the objects and the retrieval of their names (e.g., Hintz & Meyer, 2015;Meyer et al, 1998;Schotter, Ferreira, & Rayner, 2013; see also Coco & Keller, 2015;Henderson, Hayes, Rehrig, & Ferreira, 2018).…”

Perceptual salience influences food choices independently of health and taste preferences

“…Perceptual salience-the degree to which exogenous features (e.g., visual brightness) contrast with their surroundings-often plays a key role in biasing attention when multiple items are present (e.g., Itti & Koch, 2001;Itti, Koch, & Niebur, 1998;Maljkovic & Nakayama, 1994). In some cases, attention can even be captured by salient items automatically, regardless of an observer's intentions (e.g., Theeuwes, 1992), and this applies to eye movements as well (e.g., Theeuwes, Kramer, Hahn, & Irwin, 1998), though this is not necessarily automatic and other factors such as current top-down goals (e.g., Bacon & Egeth, 1994;Folk, Remington, & Johnston, 1992) or semantic information within visual scenes (e.g., Henderson, Hayes, Rehrig, & Ferreira, 2018) can override perceptual salience as a driver of attentional focus. Thus, the eyes are likely to spend more time on parts of a display that are perceptually salient (e.g., Parkhurst, Law, & Niebur, 2002).…”