Localizing volumetric motion for action recognition in realistic videos

Abstract

This paper presents a novel motion localization approach for recognizing actions and events in real videos. Examples include StandUp and Kiss in Hollywood movies. The challenge can be attributed to the large visual and motion variations imposed by realistic action poses. Previous works mainly focus on learning from descriptors of cuboids around space time interest points (STIP) to characterize actions. The size, shape and space-time position of cuboids are fixed without considering the underlying motion dynamics. This often results in large set of fragmentized cuboids which fail to capture long-term dynamic properties of realistic actions. This paper proposes the detection of spatio-temporal motion volumes (namely Volume of Interest, VOI) of scale and position adaptive to localize actions. First, motions are described as bags of point trajectories by tracking keypoints along the time dimension. VOIs are then adaptively extracted by clustering trajectory on the motion mainfold. The resulting VOIs, of varying scales and centering at arbitrary positions depending on motion dynamics, are eventually described by SIFT and 3D gradient features for action recognition. Comparing with fixed-size cuboids, VOI allows comprehensive modeling of long-term motion and shows better capability in capturing contextual information associated with motion dynamics. Experiments on a realistic Hollywood movie dataset show that the proposed approach can achieve 20% relative improvement compared to the state-ofthe-art STIP based algorithm. Categories and Subject Descriptors KeywordsHuman action recognition, Realistic videos, Motion subspace learning, Keypoint trajectory, Mean-shift clustering Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Action Similarity Labeling Challenge

“…In addition, the actions they include are non-periodic and well defined in time. These sets, although new, have already drawn a lot of attention (see for example [14], [28], [29], [30]). …”

High Five: Recognising human interactions in TV shows

“…Our descriptor does this by coarsely quantifying appearance and motion inside this region. This is in contrast to other approaches in single-action recognition [7,11,12,15,22], where features are estimated in the whole frame or video and then clustered to localise where the action is happening. Another advantage for implementing a personcentred descriptor is that, depending on the camera angle, both persons are not always visible in a given frame, and we would like to be able to provide a classification in these instances.…”

“…Previous work in two-person interaction recognition is scarce compared to closely related areas such as single-person action recognition [7,10,12,22], group action recognition [14,24] and human-object interaction recognition [16,23]. Closer to our work are [4,17,19], where interactions are generally recognised in a hierarchical manner putting special attention on higher level descriptions and using very constrained data.…”

“…These approaches rely heavily upon many low level image pre-processing steps like background subtraction and segmentation of body parts which are, by themselves, very difficult problems to solve when working with more complex scenarios. In contrast, recent publications on single-action recognition have shown a natural move from simplified and constrained datasets to more realistic ones [11,12,13,21,22]. One of the contributions of this paper is the compilation of a realistic human interaction dataset extracted from a collection of TV shows (Section 2).…”

Action Detection in Crowd

“…We use local features because they are better at handling occlusions and appearance changes [13] thus more suitable for crowded environments. The different categories of local features includes volumetric descriptors computed after 3D interest point detection [12,14,16] which encode information in local spatio-temporal blocks, trajectory descriptors [11,15,17] which tracks spatial (2D) interest points over time, and flow descriptors [1,4] which use dense optical flow information. In this paper we use the trajectory based descriptor similar to that of Sun et al [15], which has been shown to perform well in complex action recognition datasets such as the Hollywood dataset [9].…”