Resolving Ambiguous Hand Pose Predictions by Exploiting Part Correlations

Abstract

The positions of the hand joints are important highlevel features for hand-based human-computer interaction. We present a novel method to predict the 3D joint positions from the depth images and the parsed hand parts obtained with a pretrained classifier. The hand parts are utilized as the additional cue to resolve the multi-modal predictions produced by the previous regression-based method without increasing the computational cost significantly. In addition, we further enforce the hand motion constraints to fuse the per-pixel prediction results. The posterior distribution of the joints is formulated as a weighted Product of Experts model based on the individual pixel predictions, which is maximized via the Expectation-Maximization algorithm on a learned low dimensional space of the hand joint parameters. The experimental results show the proposed method improves the prediction accuracy considerably compared to the rivals that also regress for the joint locations from the depth images. Especially, we show that the regressor learned on synthesized dataset also gives accurate prediction on real-world depth images by enforcing the hand part correlations despite their discrepancies.

Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs

“…where Φ is constrained to take the linear form. In order to learn the low dimensional subspace Ω of hand configuration constrains, PCA is performed on joint locations in the training dataset [12]. E = [e 1 , e 2 , · · · , e M ] are the principal components, α = [α 1 , α 2 , · · · , α M ] T are the coefficients of the principal components, u is the empirical mean vector, and M 3 × K. As proved in the supplementary material, given the linear constrains of Φ, the optimal coefficient vector α * = [α * 1 , α * 2 , · · · , α * M ] T is:…”

Barehanded music

“…Among the discriminative methods, random regression forest and its variants have proven effective to capture hand pose in depth images [Xu and Cheng 2013;Tang et al 2013;Liang et al 2015;Sun et al 2015]. In [Xu and Cheng 2013], it is used to regress for hand joint angles directly.…”

Egocentric hand pose estimation and distance recovery in a single RGB image

“…As the semantics of the body or hand parts prove helpful for pose estimation [12,13], we propose to derive the semantic hand parts from the silhouette as intermediate features to predict Φ, as shown in Fig. 4.…”

“…Technically, we utilize the Conditional Regression Forest (CRF) [11] to predict the hand pose and the hand distance jointly with a binary context descriptor, and the hand distance is modeled as a hidden variable for inference. Moreover, motivated by the previous work [12,13] on two-stage pose inference with classified body parts, we propose to first extract the semantic hand parts from the hand silhouette with a Random Decision Forest (RDF) classifier [14], and use them as the features for pose estimation. This proves to improve the accuracy considerably.…”