Automatic Segmentation and Recognition of Human Actions in Monocular Sequences

Abstract

This paper addresses the problem of silhouettebased human action segmentation and recognition in monocular sequences. Motion History Images (MHIs), used as 2D templates, capture motion information by encoding where and when motion occurred in the images. Inspired by codebook approaches for object and scene categorization, we first construct a codebook of temporal motion templates by clustering all the MHIs of each particular action. These MHIs capture different actors, speeds and a wide range of camera viewpoints. In this paper, we use a Kohonen's Self-Organizing Map (SOM) to simultaneously cluster the MHI templates and represent them in lower dimensional subspaces. To cope with temporal segmentation, and concurrently carry out action recognition, a new architecture is proposed where the obsrvation MHIs are projected onto all these actionspecific manifolds and the Euclidean distance between each MHI and the nearest cluster within each action-manifold constitutes the observation vector of a Markov Model. To estimate the state/action at each time step, we introduce a new method based on Observable Markov Models (OMMs) where the Markov model is augmented with a neutral state. The combination of our action-specific manifolds with the augmented OMM allows to automatically segment and recognize long sequences of consecutive actions, without any prior knowledge about initial and ending frames of each action. Importantly, our method allows to interpolate betweeen training viewpoint and recognizes actions, independently of the camera viewpoint, even from unseen viewpoints.

Learning a Mid-Level Representation for Multiview Action Recognition

“…Accuracy Singh et al [31] 61.8% Orrite et al [32] 75.0% Cheema et al [33] 75.5% Murtaza et al [34] 81.6% Our method 91.2% information gain of data split. More concretely, denotes the probability that the action recognition task is selected at each node.…”

“…Since our method takes humancentered subvolumes recorded from multiple views as input, , , } ∈cub( ),V=1: , =1: ; Predefined parameters dep max , ∈ (0, 1), ∈ (0, 1), and ∈ (0, 1); Output: Decision tree Tree , ; (1) Build a bootstrap dataset , by random sampling from with replacement; (2) Create a root node and set its depth to 1, then assign all cuboids in , to it; (3) Initialize an unsettled node queue Υ = 0 and push the root node into Υ; (4) while Υ ̸ = 0 do (5) Pop the first node in Υ; (6) if depth of is larger than dep max or cuboids assigned to belong to the same action and position then (7) Label node as a leaf, and then calculate P and Q from cuboids at node ; (8) Add a triple ( , , ) into decision tree Tree , ; (9) else (10) Initialize the feature candidate set Δ = 0; (11) if random number < then (12) Add a set of randomly selected optical flow features to Δ; (13) else (14) Add a set of randomly selected HOG3D features to Δ; (15) end if (16) if random number < then (17) Add two-dimensional temporal context features to Δ; (18) end if (19) maxgain = −∞, generate a random number ; (20) for each ∈ Δ do (21) if < then (22) Search for the corresponding threshold and compute information gain ( ) in terms of action labels of cuboids arriving at ; (23) else (24) Search for the corresponding threshold and compute information gain ( ) in terms of positions of cuboids arriving at ; (25) end if (26) if ( ) > maxgain then (27) * = , * = ; (28) end if (29) end for (30) Create left children node and right children node , set their depth to dep + 1, and assign each cuboid arriving at to or according to * and * ; then push node * and * into Υ; (31) Add a quintuple ( , , , * , * ) into decision tree Tree , ; (32) end if (33) end while (34) return Decision tree Tree , ; Algorithm 1: Construction of a decision tree.…”

Multi-view Human Action Recognition Using Histograms of Oriented Gradients (HOG) Description of Motion History Images (MHIs)

“…The second direction is based on multi-view learning during training and testing of unknown action is done based on these learned features. As in [21]- [23] no feature fusion is used therefore there is no need to have all camera views available during training stage. The advantage of their approach is that they can handle missing views of an action.…”

PMHI: Proposals From Motion History Images for Temporal Segmentation of Long Uncut Videos

“…end for 5: % after above loop 6: end for Uncut silhouette video τ (1) τ (2) τ (3) … τ (w) Generating Motion History Images (MHIs)…”

“…Most existing methods exhaustively apply an action classifier at every frame in a sliding window fashion for video segmentation [2][3][4][5][6]. These approaches are computationally expensive for the analysis of large-scale videos.…”