Fusing multi-modal features for gesture recognition

Wu, Jiaxiang; Cheng, Jian; Zhao, Chaoyang; Lu, Hanqing

doi:10.1145/2522848.2532589

Cited by 58 publications

(38 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…And our model achieves better recognition rate than the winner of the challenge [29] that uses variant of nearest neighbour and dynamic time warping in the ChaLearn Gesture dataset.…”

Section: Methodsmentioning

confidence: 90%

Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition

Shao

2014

2014 IEEE Conference on Computer Vision and Pattern Recognition

194

147

View full text Add to dashboard Cite

Over the last few years, with the immense popularity of the Kinect, there has been renewed interest in developing methods for human gesture and action recognition from 3D skeletal data. A number of approaches have been proposed to extract representative features from 3D skeletal data, most commonly hard wired geometric or bio-inspired shape context features. We propose a hierarchial dynamic framework that first extracts high level skeletal joints features and then uses the learned representation for estimating emission probability to infer action sequences. Currently gaussian mixture models are the dominant technique for modeling the emission distribution of hidden Markov models. We show that better action recognition using skeletal features can be achieved by replacing gaussian mixture models by deep neural networks that contain many layers of features to predict probability distributions over states of hidden Markov models. The framework can be easily extended to include a ergodic state to segment and recognize actions simultaneously.

show abstract

“…And our model achieves better recognition rate than the winner of the challenge [29] that uses variant of nearest neighbour and dynamic time warping in the ChaLearn Gesture dataset.…”

Section: Methodsmentioning

confidence: 90%

Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition

Shao

2014

2014 IEEE Conference on Computer Vision and Pattern Recognition

194

147

View full text Add to dashboard Cite

show abstract

“…We follow the experimental protocol adopted in [3,14,22,25] and provide precision, recall and F1-score measures on the validation set. We compare our model with Yao et al [25], Wu et al [22], Pfister et al [14], and Fernando et al [3].…”

Section: Methodsmentioning

confidence: 99%

Skeleton based action recognition with convolutional neural network

Wang

2015

2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)

359

244

View full text Add to dashboard Cite

Temporal dynamics of postures over time is crucial for sequence-based action recognition. Human actions can be represented by the corresponding motions of articulated skeleton. Most of the existing approaches for skeleton based action recognition model the spatial-temporal evolution of actions based on hand-crafted features. As a kind of hierarchically adaptive filter banks, Convolutional Neural Network (CNN) performs well in representation learning. In this paper, we propose an end-to-end hierarchical architecture for skeleton based action recognition with CNN. Firstly, we represent a skeleton sequence as a matrix by concatenating the joint coordinates in each instant and arranging those vector representations in a chronological order. Then the matrix is quantified into an image and normalized to handle the variable-length problem. The final image is fed into a CNN model for feature extraction and recognition. For the specific structure of such images, the simple max-pooling plays an important role on spatial feature selection as well as temporal frequency adjustment, which can obtain more discriminative joint information for different actions and meanwhile address the variable-frequency problem. Experimental results demonstrate that our method achieves the state-of-art performance with high computational efficiency, especially surpassing the existing result by more than 15 percentage on the challenging ChaLearn gesture recognition dataset.

show abstract

“…Precision Recall F-score Pfister et al [17] 61.2% 62.3% 61.7% Yao et al [28] --56.0% Wu et al [26] 59.9% 59.3% 59.6% VideoDarwin [5] 74.0% 73.8% 73.9% HiVideoDarwin 74.9% 75.6% 74.6% Table 3. Statistical analysis for parameters.…”

Section: Approachmentioning

confidence: 99%

Hierarchical motion evolution for action recognition

Wang

2015

2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)

View full text Add to dashboard Cite

Human action can be decomposed into a series of temporally correlated motions. Since the traditional bag-of-wordsframework based on local features cannot model global motion evolution of actions, models like Recurrent Neural Network (RNN) [15] and VideoDarwin [5] are accordingly explored to capture video-wise temporal information. Inspired by VideoDarwin, in this paper, we present a novel hierarchical scheme to learn better video representation, called HiVideoDarwin. Specifically, we first use different ranking machines to learn motion descriptors of local video clips. Then, in order to model motion evolution, we encode features obtained in previous layer again using a ranking machine. Compared with VideoDarwin, HiVideoDarwin captures the global and high-level video representation and is robust to large appearance changes. Compared with RN-N, HiVideoDarwin can also abstract semantic information in a hierarchical way and is fast to compute and easy to interpret. We evaluate the proposed method on two datasets, namely MPII Cooking and Chalearn. Experimental results show that HiVideoDarwin has distinct advantages over the state-of-the-art models. Additional sensitivity analysis reveals that the overall results are hardly affected by parameter changes.

show abstract

Fusing multi-modal features for gesture recognition

Cited by 58 publications

References 20 publications

Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition

Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition

Skeleton based action recognition with convolutional neural network

Hierarchical motion evolution for action recognition

Contact Info

Product

Resources

About