Unified Image and Video Saliency Modeling

Abstract

Visual saliency modeling for images and videos is treated as two independent tasks in recent computer vision literature. On the one hand, image saliency modeling is a well-studied problem and progress on benchmarks like SALICON and MIT300 is slowing. For video saliency prediction on the other hand, rapid gains have been achieved on the recent DHF1K benchmark through network architectures that are optimized for this task. Here, we take a step back and ask: Can image and video saliency modeling be approached via a unified model, with mutual benefit? We find that it is crucial to model the domain shift between image and video saliency data and between different video saliency datasets for effective joint modeling. We identify different sources of domain shift and address them through four novel domain adaptation techniques-Domain-Adaptive Priors, Domain-Adaptive Fusion, Domain-Adaptive Smoothing and Bypass-RNN-in addition to an improved formulation of learned Gaussian priors. We integrate these techniques into a simple and lightweight encoder-RNN-decoder-style network, UNISAL, and train the entire network simultaneously with image and video saliency data. We evaluate our method on the video saliency datasets DHF1K, Hollywood-2 and UCF-Sports, as well as the image saliency datasets SALICON and MIT300. With one set of parameters, our method achieves state-of-the-art performance on all video saliency datasets and is on par with the state-of-the-art for image saliency prediction, despite a 5 to 20-fold reduction in model size and the fastest runtime among all competing deep models. We provide retrospective analyses and ablation studies which demonstrate the importance of the domain shift modeling. The code is available at https://github.com/rdroste/unisal.

Hierarchical Domain-Adapted Feature Learning for Video Saliency Prediction

“…While unsupervised domain adaptation has been applied to image classification (Ganin 2016;Tzeng et al 2017), face recognition (Kan et al 2015), object detection (Tang 2016), semantic segmentation (Zhang et al 2020) and video action recognition (Li et al 2018) (among others), our work is, to our knowledge, the first to deal with unsupervised domain adaptation on video saliency prediction. It is worthwhile to note that this is technically and fundamentally different from the form of domain adaptation proposed in UNISAL (Droste et al 2020), that, instead, learns domain-specific parameters. This means that, at inference time, UNISAL requires to know the source dataset of a given input in order to select domain-specific learned parameters.…”

“…It is also different from unsupervised salient object detection (Zhang et al 2018), which, instead, attempts to predict saliency by exploiting large unlabelled or weakly-labelled samples. However, we also provide HD 2 S with domain-specific learning capabilities as in (Droste et al 2020), showing how this mechanism improves performance but cannot be applied in unsupervised domain adaptation scenarios.…”

“…In certain multi-source training scenarios (e.g., as done in (Droste et al 2020)), one may assume that annotations are available for all employed datasets, thus enabling supervised training on all of them. When applying our saliency prediction model to this scenario, we provide it with domainspecific operations (Droste et al 2020), which address the domain shift among different datasets.…”

Bio-Inspired Representation Learning for Visual Attention Prediction

“…Fourthly, by employing an element-wise multiplication between S m V,C and S is further processed with the 1 × 1 convolution and resize-convolution operation to generate a high-level semantic-aware attention map F m semantic . 2) Center-Bias Prior: According to previous studies [28], [66], [67], human attention tends to concentrate on the center of scenes, which is termed as center-bias phenomenon. To this end, a learnable center-bias prior function is adopted according to our preceding work [40].…”

Glimpse: A Gaze-Based Measure of Temporal Salience

“…Salience in dynamic scenes is related to but conceptually different from salience in static images [ 27 ]. Specific methods for the dynamic case have been studied [ 28 , 29 , 30 , 31 , 32 , 33 ] and, very recently, unified image-video approaches [ 34 ] proposed, but only in the context of spatial salience. For gaze prediction, temporal features are found to be of key importance in rare events, so spatial static features can explain gaze in most cases [ 35 ].…”