Video event detection and summarization using audio, visual and text saliency

Abstract

Detection of perceptually important video events is formulated here on the basis of saliency models for the audio, visual and textual information conveyed in a video stream. Audio saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color and motion. Text saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The various modality curves are integrated in a single attention curve, where the presence of an event may be signified in one or multiple domains. This multimodal saliency curve is the basis of a bottom-up video summarization algorithm, that refines results from unimodal or audiovisual-based skimming. The algorithm performs favorably for video summarization in terms of informativeness and enjoyability.

Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

“…Finally, the selected clips are tailored together using overlapadd (fade-in fade-out) for both the audio and visual streams. More details are provided in [13].…”

“…Computational models of single-and multimodal saliency [10]- [13] have been applied to emerging multimedia applications such as automatic video abstraction, summarization, indexing and browsing [14], [15]. Summarization refers to producing a shorter, in duration, version of a video that contains essential information for content understanding, without sacrificing much of the original's informative, functional or aesthetical purpose.…”

Spatiotemporal Features for Action Recognition and Salient Event Detection

“…They use motion, face, and camera attention along with audio attention models (audio saliency and speech/music) as cues to capture salient information and identify the audio and video segments to compose the summary. In a similar fashion, Evangelopoulos et al [42,43] and Rapantzikos et al [44] fuse audio, visual, and text saliency measures into a single attention curve and select prominent parts of this measure to generate a summary.…”

Vision, Attention Control, and Goals Creation System

“…They use motion, face, and camera attention along with audio attention models (audio saliency and speech/music) as cues to capture salient information and identify the audio and video segments to compose the summary. Rapantzikos et al (Evangelopoulos et al 2008(Evangelopoulos et al , 2009) build further on visual, audio, and textual attention models for visual summarization. The authors form a multimodal saliency curve integrating the aural, visual, and textual streams of videos based on efficient audio, image, and language processing and employ it as a metric for video event detection and abstraction.…”