TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection

Abstract

TASED-Net is a 3D fully-convolutional network architecture for video saliency detection. It consists of two building blocks: first, the encoder network extracts lowresolution spatiotemporal features from an input clip of several consecutive frames, and then the following prediction network decodes the encoded features spatially while aggregating all the temporal information. As a result, a single prediction map is produced from an input clip of multiple frames. Frame-wise saliency maps can be predicted by applying TASED-Net in a sliding-window fashion to a video. The proposed approach assumes that the saliency map of any frame can be predicted by considering a limited number of past frames. The results of our extensive experiments on video saliency detection validate this assumption and demonstrate that our fully-convolutional model with temporal aggregation method is effective. TASED-Net significantly outperforms previous state-of-the-art approaches on all three major large-scale datasets of video saliency detection: DHF1K, Hollywood2, and UCFSports. After analyzing the results qualitatively, we observe that our model is especially better at attending to salient moving objects.

Advances in Video Compression System Using Deep Neural Network: A Review and Case Studies

“…Earlier attempts to predict saliency typically utilized handcrafted image features, such as color, intensity, and orientation contrast [39], motion contrast [40], and camera motion [41]. Later on, DNN-based semantic-level features were extensively investigated for both image content [42]- [48] and video sequences [49]- [55]. Among these features, image saliency prediction only exploits spatial information, while video saliency prediction often relies on spatial and temporal attributes jointly.…”

A Novel Spatio-Temporal 3D Convolutional Encoder-Decoder Network for Dynamic Saliency Prediction

“…The other is the ineffective structures in extracting joint spatio-temporal FIGURE 1. Cases that TASED-Net [16] fails to allocate saliency and predict shifts of saliency focus. The rows from top to bottom are the input, ground-truth, and saliency maps predicted by TASED-Net [16].…”

“…Deep models based on LSTM or its variants [7]- [15] are difficult to train for long-range temporal evolution due to the frame-by-frame data feeding scheme. Models based on 3D fully CNN (3DCNN) [16], [17] still may fail to allocate saliency or to predict shifts of saliency focus when there are multiple instances.…”

“…Taking TASED-Net [16] as an example, which is based on 3DCNN and achieves the state-of-the-art performance, it still may fail in both cases mentioned above, as shown in Fig. 1.…”

“…In the first frame, a woman running towards camera attracts most attention according to ground-truth. However, TASED-Net [16] allocates nearly equal saliency to the woman and a parked car. The other four frames show a shift of saliency focus from a standing veteran to a young man.…”