STAViS: Spatio-Temporal AudioVisual Saliency Network

Tsiami, Antigoni; Koutras, Petros; Maragos, Petros

doi:10.1109/cvpr42600.2020.00482

Cited by 66 publications

(46 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While in this paper, we mainly focus on end-to-end deep learning based audio-visual processing model. Recent years have seen some good works on audio-visual saliency estimation or prediction by using deep learning-based end-to-end models, such as STAVIS [14]. STAVIS shares similar research objectives with this paper.…”

Section: Audio-visual Saliency Modelsmentioning

confidence: 74%

Deep Audio-Visual Fusion Neural Network for Saliency Estimation

Yao

Min

Zhai

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

In this work, we propose a deep audio-visual fusion model to estimate the saliency of videos. The model extracts visual and audio features with two separate branches and fuses them to generate the saliency map. We design a novel temporal attention module to utilize the temporal information and a spatial feature pyramid module to fuse the spatial information. Then a multi-scale audio-visual fusion method is used to integrate different modalities. Furthermore, we propose a new dataset for audio-visual saliency estimation. The proposed dataset consists of 202 high quality video squences with a large range of motions, scenes and object types. Many of the videos have high audio-visual correspondence. Several experiments are conducted on different datasets. The results demonstrate that our model outperforms the previous state-of-the-art methods by a large margin and the proposed dataset can serve as a new benchmark for the audio-visual saliency estimation task.

show abstract

Section: Audio-visual Saliency Modelsmentioning

confidence: 74%

Deep Audio-Visual Fusion Neural Network for Saliency Estimation

Yao

Min

Zhai

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…Each video frame is resized at 112×112 pixels. Following the previous work [59], the data augmentation is also employed for random generation of training samples. The implementation adopts the 3D ResNet-50 [24] as backbone for encoding spatio-temporal visual features, and applies SoundNet [23] as backbone for encoding high-level audio semantic features.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…TASED [78] exploits 3D fully-convolutional network architecture to generate the visual attention map of each frame by considering several past frames. STAViS [59] combines both visual and auditory information for VAP in videos.…”

Section: Comparison With State-of-the-artsmentioning

confidence: 99%

Bio-Inspired Representation Learning for Visual Attention Prediction

Yuan

Ning

2021

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Visual Attention Prediction (VAP) is a significant and imperative issue in the field of computer vision. Most of existing VAP methods are based on deep learning. However, they do not fully take advantage of the low-level contrast features while generating the visual attention map. In this paper, a novel VAP method is proposed to generate visual attention map via bioinspired representation learning. The bio-inspired representation learning combines both low-level contrast and high-level semantic features simultaneously, which are developed by the fact that human eye is sensitive to the patches with high contrast and objects with high semantics. The proposed method is composed of three main steps: 1) feature extraction, 2) bio-inspired representation learning and 3) visual attention map generation. Firstly, the high-level semantic feature is extracted from the refined VGG16, while the low-level contrast feature is extracted by the proposed contrast feature extraction block in a deep network. Secondly, during bio-inspired representation learning, both the extracted low-level contrast and high-level semantic features are combined by the designed densely connected block, which is proposed to concatenate various features scale by scale. Finally, the weightedfusion layer is exploited to generate the ultimate visual attention map based on the obtained representations after bio-inspired representation learning. Extensive experiments are performed to demonstrate the effectiveness of the proposed method.

show abstract

“…More recently, there have been some efforts related to audiovisual saliency [60][61][62][63], but very few explicitly incorporate spatial (two-dimensional) audio [62]. However, some of the drawbacks of those models are that they are not based on the neural principles and do not use spatial audio which is an important feature in the auditory domain.…”

Section: Related Workmentioning

confidence: 99%

A Biologically Motivated, Proto-Object-Based Audiovisual Saliency Model

Ramenahalli

2020

View full text Add to dashboard Cite

The natural environment and our interaction with it are essentially multisensory, where we may deploy visual, tactile and/or auditory senses to perceive, learn and interact with our environment. Our objective in this study is to develop a scene analysis algorithm using multisensory information, specifically vision and audio. We develop a proto-object-based audiovisual saliency map (AVSM) for the analysis of dynamic natural scenes. A specialized audiovisual camera with 360∘ field of view, capable of locating sound direction, is used to collect spatiotemporally aligned audiovisual data. We demonstrate that the performance of a proto-object-based audiovisual saliency map in detecting and localizing salient objects/events is in agreement with human judgment. In addition, the proto-object-based AVSM that we compute as a linear combination of visual and auditory feature conspicuity maps captures a higher number of valid salient events compared to unisensory saliency maps. Such an algorithm can be useful in surveillance, robotic navigation, video compression and related applications.

show abstract

STAViS: Spatio-Temporal AudioVisual Saliency Network

Cited by 66 publications

References 54 publications

Deep Audio-Visual Fusion Neural Network for Saliency Estimation

Deep Audio-Visual Fusion Neural Network for Saliency Estimation

Bio-Inspired Representation Learning for Visual Attention Prediction

A Biologically Motivated, Proto-Object-Based Audiovisual Saliency Model

Contact Info

Product

Resources

About