Review of Visual Saliency Prediction: Development Process from Neurobiological Basis to Deep Models

Abstract

The human attention mechanism can be understood and simulated by closely associating the saliency prediction task to neuroscience and psychology. Furthermore, saliency prediction is widely used in computer vision and interdisciplinary subjects. In recent years, with the rapid development of deep learning, deep models have made amazing achievements in saliency prediction. Deep learning models can automatically learn features, thus solving many drawbacks of the classic models, such as handcrafted features and task settings, among others. Nevertheless, the deep models still have some limitations, for example in tasks involving multi-modality and semantic understanding. This study focuses on summarizing the relevant achievements in the field of saliency prediction, including the early neurological and psychological mechanisms and the guiding role of classic models, followed by the development process and data comparison of classic and deep saliency prediction models. This study also discusses the relationship between the model and human vision, as well as the factors that cause the semantic gaps, the influences of attention in cognitive research, the limitations of the saliency model, and the emerging applications, to provide new saliency predictions for follow-up work and the necessary help and advice.

Low-level factors increase gaze-guidance under cognitive load: A comparison of image-salience and semantic-salience models

“…A model for visual saliency was originally proposed by Itti, Koch, & Neibur [ 12 ] in which local center-surround difference maps of linear filter responses to color, intensity and orientation are combined and weighted to generate local feature salience maps, the peaks of which are sequentially fixated. Many elaborations of this general approach have now been proposed, for recent review see [ 13 , 14 ]. In this paper, we employ graph-based visual saliency (GBVS), which is one such example that combines three main low-level features: color, edge-orientation, and intensity [ 2 ].…”

Visual Search Patterns During Exploration of Naturalistic Scenes are Driven by Saliency Cues in Individuals with Cerebral Visual Impairment

“…While participants had previous knowledge of approximately what a desk would look like and where it is likely to appear 45,46 they did not have any knowledge of what the particular desk they were searching for looked like exactly. This could have led to xations being directed at any visually salient feature rst, (as visually salient features are known to capture gaze 33,34 , until the target was found. Meaning, without available image features to search for, gaze was more guided by image salience than by image semantics in control participants.…”

“…[4][5][6][7] ). Based on this evidence, models have been developed that predict gaze behavior driven by image salience (for review, see 33,34 ), where areas of a scene labeled as having high image salience have a correspondingly higher likelihood of xation. Image semantics are computed from higher-level top-down factors based on prior knowledge.…”

Driver Attention Prediction based on Multi-ScaleFeature Fusion

“…Visual attention prediction is a task that mimics the mechanisms of human visual attention, which involves knowledge not only from computer vision, but also from neurobiology and psychology [13]. This task has attracted considerable interest from computer vision researchers and aims to generate saliency maps by predicting the regions within an image or video that are most likely to attract attention [14] [15].…”