Action Recognition Robust to Background Clutter by Using Stereo Vision

Abstract

Abstract. An action recognition algorithm which works with binocular videos is presented. The proposed method uses standard bag-of-words approach, where each action clip is represented as a histogram of visual words. However, instead of using classical monocular HoG/HoF features, we construct features from the scene-flow computed by a matching algorithm on the sequence of stereo images. The resulting algorithm has a comparable or slightly better recognition accuracy than standard monocular solution in controlled setup with a single actor present in the scene. However, we show its significantly improved performance in case of strong background clutter due to other people freely moving behind the actor.

Exploiting stereoscopic disparity for augmenting human activity recognition performance

“…This intuition explains the gap in classification rate between methods E r and D r : with E r no such filtering takes place and the modest gains in mean average precision, in comparison to the monocular approach, may simply be attributed to the more dense video description, since E r i = C d i ∪ C r i . It also confirms the conclusions reached in [28], regarding the use of stereoscopic data to exploit video background-foreground segmentation for activity recognition. However, contrary to [28], the proposed method D r operates along these lines only implicitly, through increasing texture invariance and scene geometry content of the video description, as well as in a generic manner, not associated with any specific feature descriptor.…”

“…It also confirms the conclusions reached in [28], regarding the use of stereoscopic data to exploit video background-foreground segmentation for activity recognition. However, contrary to [28], the proposed method D r operates along these lines only implicitly, through increasing texture invariance and scene geometry content of the video description, as well as in a generic manner, not associated with any specific feature descriptor.…”

“…This outcome is compatible with the intuition that, in most videos, the depicted activity is most likely located at a medium distance from the camera during video acquisition. Additionally, the potential of stereoscopic data in segmenting background from foreground for enhancing activity recognition performance, as indicated in [28], is validated once more. Table IV shows the average precision measured per action class, for the typical, monocular BoF representation method (D = 1, i = 0), the best-performing stereoscopic schemes (D = 3, i = 2 and D = 4, i = 3) and the best method reported in [24].…”

“…It is an active research field at the intersection of computer vision, pattern recognition and machine learning, where significant progress has been made during the last decade. Depending on the application scenario, several approaches have been proposed, ranging from the recognition of simple human actions in constrained environments [1], [2], [3], [4], to the recognition of complex actions (also referred to as activities) in unconstrained environments [5] [6] [7] [8] [9] [10].…”

Audio-visual robot command recognition

“…The first scheme is based on a bagof-words (BoW) approach while the second one incorporates temporal structure by means of HMMs. Scene-flow [15] and STIPs [8] are used as visual features and MFCC as auditory features. The fusion is done at the classification stage through a modality-weighting scheme, i.e., pooling.…”