Short-term audio-visual atoms for generic video concept classification

Abstract

We investigate the challenging issue of joint audio-visual analysis of generic videos targeting at semantic concept detection. We propose to extract a novel representation, the Short-term Audio-Visual Atom (S-AVA), for improved concept detection. An S-AVA is defined as a short-term region track associated with regional visual features and background audio features. An effective algorithm, named ShortTerm Region tracking with joint Point Tracking and Region Segmentation (STR-PTRS), is developed to extract S-AVAs from generic videos under challenging conditions such as uneven lighting, clutter, occlusions, and complicated motions of both objects and camera. Discriminative audio-visual codebooks are constructed on top of S-AVAs using Multiple Instance Learning. Codebook-based features are generated for semantic concept detection. We extensively evaluate our algorithm over Kodak's consumer benchmark video set from real users. Experimental results confirm significant performance improvements -over 120% MAP gain compared to alternative approaches using static region segmentation without temporal tracking. The joint audio-visual features also outperform visual features alone by an average of 8.5% (in terms of AP) over 21 concepts, with many concepts achieving more than 20%. Categories and Subject Descriptors General Terms Algorithms, Experimentation KeywordsSemantic concept detection, joint audio-visual analysis, shortterm audio-visual atom, audio-visual codebook Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MM '09, October 19-24, 2009, Beijing, China. Copyright 2009 track of a birthday cake and the background birthday music form a salient audio-visual cue to describe "birthday" videos. The short-term region track of a horse and the background horse running footstep sound give a salient audio-visual cue for the "animal" concept.

Semantic Model Vectors for Complex Video Event Recognition

“…Zhou et al [64] use Gaussian mixture models (GMM) instead of the standard BoW approach to describe an event as a SIFT-Bag. Jiang et al [19] follow the same direction by defining an event as a combination of short-term audio-visual atoms. Nevertheless, valuable spatial context is neglected due to the spatial structure-free BoW representation, which limits the potential of these methods.…”

Spatio-temporal information for human action recognition

“…Jiang et al [10] grouped visual and audio features together with their temporal relationship and computed combined features from these groups. Similarly, Jhou et al [13] constructed a bigraph with temporal concurrency between visual words and employed a k-way segmentation algorithm to combine visual and audio features.…”

“…Researches have proposed several approaches to make further use of the different information in videos. Jiang et al [10] introduced an audio-visual atom as joint audio-visual feature for video concept recognition. Jiang and Loui [11] used the temporal relationship between audio feature and visual feature to group clusters up, and then construct new features from the groups.…”

Benchmarking methods for audio-visual recognition using tiny training sets

“…Early Fusion: In [1] an audio-visual representation named short-term audio-visual atom is proposed. It is a concatenation of color/texture, motion and auditory features.…”