Hierarchical Domain-Adapted Feature Learning for Video Saliency Prediction

Abstract

In this work, we propose a 3D fully convolutional architecture for video saliency prediction that employs hierarchical supervision on intermediate maps (referred to as conspicuity maps) generated using features extracted at different abstraction levels. We provide the base hierarchical learning mechanism with two techniques for domain adaptation and domain-specific learning. For the former, we encourage the model to unsupervisedly learn hierarchical general features using gradient reversal at multiple scales, to enhance generalization capabilities on datasets for which no annotations are provided during training. As for domain specialization, we employ domain-specific operations (namely, priors, smoothing and batch normalization) by specializing the learned features on individual datasets in order to maximize performance. The results of our experiments show that the proposed model yields state-of-the-art accuracy on supervised saliency prediction. When the base hierarchical model is empowered with domain-specific modules, performance improves, outperforming state-of-the-art models on three out of five metrics on the DHF1K benchmark and reaching the second-best results on the other two. When, instead, we test it in an unsupervised domain adaptation setting, by enabling hierarchical gradient reversal layers, we obtain performance comparable to supervised state-of-the-art. Source code, trained models and example outputs are publicly available at https://github.com/perceivelab/hd2s.

TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillation

“…UNISAL [6] proposes a multi-objective unified framework for both 2D and 3D saliency with domain-specific modules and a lightweight recurrent architecture to handle temporal dynamics; While single-decoder approaches are common, multi-decoder output integration has recently attracted interest. DVA [35] and HD2S [1] fuse maps predicted by independent decoders operating at different abstraction levels. RecSal [30] predicts multiple saliency maps in a multiobjective training framework.…”

“…Many solutions have been proposed, based on different assumptions on how to capture video saliency. It is interesting to note that, in spite of the remarkably different research directions followed by the variety of works in the literature, top results over video saliency prediction benchmarks are very close [1,3,36], suggesting that predictions of different models are similar. We assessed the validity of this conclusion by comparing three of the best performing methods on the DHF1K dataset [36] -TASED [25], HD2S [1] and ViNet [16] -not in terms of their scores on summary metrics, but in terms of the relative similarity of the predicted saliency maps.…”

Video Content Analysis of Human Sports under Engineering Management Incorporating High-Level Semantic Recognition Models

“…Consider that optical flow can be used to direct human foreground attention when appropriate compensation is applied to the movement of the lens. We investigate the combination of spatial streaming embedding CNN and temporal streaming CNN to form a dual-stream convolutional neural network to learn video features [ 20 ]. The purpose of introducing an optical flow attention layer from the temporal network to the spatial network is to guide the spatial flow to pay more attention to the human foreground region and to reduce the effect of background noise.…”

Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos

“…Visual saliency prediction. Visual saliency models have been widely developed to predict where people look in images (Huang et al, 2015;Zhang and Sclaroff, 2016;Pan et al, 2017;Wang and Shen, 2017;Cornia et al, 2018;Li et al, 2014) or videos (Hossein Khatoonabadi et al, 2015;Bak et al, 2017;Liu et al, 2017;Jiang et al, 2021;Wang et al, 2018;Min and Corso, 2019;Zanca et al, 2019;Bellitto et al, 2021;Li et al, 2010;Souly and Shah, 2016). The seminal work of Itti et al (1998) proposed a computational model to predict the image saliency, via combining three low-level features including color, intensity, and orientation.…”