Saliency Prediction in Uncategorized Videos Based on Audio-Visual Correlation

Abstract

Substantial research has been done in saliency modeling to make intelligent machines that can perceive and interpret their surroundings and focus only on the salient regions in a visual scene. But existing spatio-temporal saliency models either treat videos as merely image sequences excluding any audio information or are unable to cope with inherently varying content. Based on the hypothesis that an audiovisual saliency model will perform better than traditional spatio-temporal saliency models, this work aims to provide a generic preliminary audio/video saliency model. This is achieved by augmenting visual saliency map with an audio saliency map computed by synchronizing low-level audio and visual features. The proposed model was evaluated using different criteria against eye fixations data for a publicly available video dataset DIEM. The evaluation results show that the model outperforms two state-of-the-art visual spatio-temporal saliency models supporting our hypothesis that an audiovisual model performs better in comparison to a visual model for natural uncategorized videos.Interestingly the effect of audio stimuli is proved to be relevant to human eye movements, for instance in [16] the authors find eye movements to be spatially biased towards the source of audio by performing an eye tracking experiment on images with spatially localized sound sources in three conditions: auditory (A), visual (V) and audio-visual (AV). Moreover, another study [17] analyzed the effects of different type of sounds on human gaze involving an experiment with 36 participants and thirteen sound classes under audio-visual and visual conditions. The sound classes are further clustered into on-screen with one sound source,