EML-NET: An Expandable Multi-Layer NETwork for saliency prediction

Jia, Sen; Bruce, Neil D. B.

doi:10.1016/j.imavis.2020.103887

Cited by 131 publications

(112 citation statements)

References 51 publications

(121 reference statements)

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…The applications of CNNs in salience detection are extensive. Some of the examples of these applications are Deep Gaze 2 [141], which presents two models for fixation predictions and object recognition; EML-NET [142] for salience feature detection using a modular approach; and DeepFix [143], which is sensitive to semantic information at different scales while using large receptive fields for global context analysis. The scope of existing models in the field is so broad that it is impossible to cover all of them in this review, but their accuracy in predicting areas of spatial salience is without question [144][145][146][147][148][149].…”

Section: Deep Learning Classifiersmentioning

confidence: 99%

Salience Models: A Computational Cognitive Neuroscience Review

Krasovskaya

MacInnes

2019

Vision

View full text Add to dashboard Cite

The seminal model by Laurent Itti and Cristoph Koch demonstrated that we can compute the entire flow of visual processing from input to resulting fixations. Despite many replications and follow-ups, few have matched the impact of the original model-so what made this model so groundbreaking? We have selected five key contributions that distinguish the original salience model by Itti and Koch; namely, its contribution to our theoretical, neural, and computational understanding of visual processing, as well as the spatial and temporal predictions for fixation distributions. During the last 20 years, advances in the field have brought up various techniques and approaches to salience modelling, many of which tried to improve or add to the initial Itti and Koch model. One of the most recent trends has been to adopt the computational power of deep learning neural networks; however, this has also shifted their primary focus to spatial classification. We present a review of recent approaches to modelling salience, starting from direct variations of the Itti and Koch salience model to sophisticated deep-learning architectures, and discuss the models from the point of view of their contribution to computational cognitive neuroscience.Vision 2019, 3, 56 2 of 24 a true sign of its importance in defining this field. We provide a review of the different approaches in modelling visual attention based on their contribution to computational cognitive neuroscience. We limit our review to models that were directly influenced by the Itti and Koch algorithm, and cover examples that model object attention, top-down influence, information theory, dual stream models, and conclude with recent advances in deep learning salience classifiers. We also include other methods of achieving the main goal: modelling image salience and the way it results in shifts of attention. The main questions this review tries to address are what contribution did these models make in our goal to explain human visual attention using computer simulations, and what directions are available for the next generation of models?In this review, we approach the salience problem from a computational cognitive neuroscience perspective, which implies a combination of theoretical and methodological concepts from several domains. Such a perspective suggests that the perfect salience model should be based on a strong theoretical foundation, model the neurobiological processes underlying visual saliency, use explicit computational tools as a means of modelling these processes, and be generative by taking both spatial and temporal predictions of visual salience into account.Studies of human visual salience have led to the creation of hundreds of computationally valid models, however, most of these models do not focus on all of the abovementioned components of salience simultaneously. We understand that with so many parameters to account for, like combining a broad cognitive theoretical scope and a focused precise neural approach, it is almost impossible to avoid trade-offs. Never...

show abstract

Section: Deep Learning Classifiersmentioning

confidence: 99%

Salience Models: A Computational Cognitive Neuroscience Review

Krasovskaya

MacInnes

2019

Vision

View full text Add to dashboard Cite

show abstract

“…Compared to visual attention models for 360 • images, those for 2D traditional images have been well developed in recent years [21], [22], [23]. Seminal methods were proposed based on low-level or high-level semantic feature extraction from handcrafted filters [24], [25], [26] or Deep Convolutional Neural Networks (DCNN) [27], [28], [29], [30], [31] thanks to the establishment of several large scale datasets [32], [33], [34]. Unfortunately, these models are not immediately usable for 360 • images because of the severe geometric distortion on the top and bottom areas in equirectangular projection.…”

Section: By Leveraging 360 • Fixation Cues and User's Orientation Sensedmentioning

confidence: 99%

A Multi-FoV Viewport-Based Visual Saliency Model Using Adaptive Weighting Losses for 360$^\circ$ Images

Chao

Zhang

Hamidouche

et al. 2021

IEEE Trans. Multimedia

View full text Add to dashboard Cite

• media allows observers to explore the scene in all directions. The consequence is that the human visual attention is guided by not only the perceived area in the viewport but also the overall content in 360 •. In this paper, we propose a method to estimate the 360 • saliency map which extracts salient features from the entire 360 • image in each viewport in three different Field of Views (FoVs). Our model is first pretrained with a largescale 2D image dataset to enable the interpretation of semantic contents, then fine-tuned with a relative small 360 • image dataset. A novel weighting loss function attached with stretch weighted maps is introduced to adaptively weight the losses of three evaluation metrics and attenuate the impact of stretched regions in equirectangular projection during training process. Experimental results demonstrate that our model achieves better performance with the integration of three FoVs and its diverse viewport images. Results also show that the adaptive weighting losses and stretch weighted maps effectively enhance the evaluation scores compared to the fixed weighting losses solutions. Comparing to other state of the art models, our method surpasses them on three different datasets and ranks the top using 5 performance evaluation metrics on the Salient360! benchmark set.

show abstract

“…1. Static saliency models: eDN [72], DeepGaze I & II [49], Mr-CNN [55], SALICON [33], DeepFix [48], SAM-ResNet [19], and EML-Net [39].…”

Section: Bottom-up Attention Modeling: Deep Learning Eramentioning

confidence: 99%

Bottom-Up Attention, Models of

Borji¹,

Tavakoli²,

Bylinskii³

2019

Encyclopedia of Computational Neuroscience

View full text Add to dashboard Cite

DefinitionAttention -a general concept covering all factors that influence selection mechanisms, whether they are scene-driven and bottom-up, or expectation-driven and top-down.Salience -parts of a stimulus (e.g. image, video, audio) that appear to an observer to stand out, relative to their neighboring parts. It is a subjective perceptual quality.Gaze -a coordinated motion of the eyes and the head that is long enough to concentrate on something. It is a key property of attention in natural behavior.Scene free-viewing -A task in which participants are asked to look at an image, without any specific instruction. As a default scene analysis task, the free viewing paradigm offers a wealth of insights regarding the cues that attract attention. Detailed Description History, scope, and organizationDeciphering the computational mechanisms by which the brain deals with the computational complexity of an overwhelmingly high volume of incoming data (at a rate of 10 8 bits/s [46]) and how the brain programs eye movements continue to be important problems in neuroscience. Where humans look in images and videos provides important clues regarding how they perceive static (still images) and dynamic scenes (videos), locate the main focus of the image, recognize actions or events, and identify the main participants.A significant amount of behavioral and computational research on attention has revealed that attention is deployed in two ways: bottom-up (BU) and top-down (TD). The bottom-up component of attention, a.k.a endogenous or stimulus-driven, processes sensory information primarily in a feed-forward manner. Typically, a series of successive transformations are applied to the input visual signal to highlight the most interesting, important, conspicuous, or so-called salient regions [45,36]. In contrast, in top-down attention, a.k.a context-driven, or goal-driven, information related to the ongoing behavior, task, or goal is selected (e.g. staying within the road lanes while driving [52]). The reader is referred to [51,2,27,57,11] for reviews of top-down attention studies. Please see also the chapter on top-down attention in this encyclopedia [66].Conventionally, bottom-up attention models generate a 2D topographic saliency map, where a value at every location determines how salient that location is, relative to its neighbors. The goal in saliency *

show abstract

EML-NET: An Expandable Multi-Layer NETwork for saliency prediction

Cited by 131 publications

References 51 publications

Salience Models: A Computational Cognitive Neuroscience Review

Salience Models: A Computational Cognitive Neuroscience Review

A Multi-FoV Viewport-Based Visual Saliency Model Using Adaptive Weighting Losses for 360$^\circ$ Images

Bottom-Up Attention, Models of

Contact Info

Product

Resources

About