Hand3D: Hand Pose Estimation using 3D Neural Network

Abstract

We propose a novel 3D neural network architecture for 3D hand pose estimation from a single depth image. Different from previous works that mostly run on 2D depth image domain and require intermediate or post process to bring in the supervision from 3D space, we convert the depth map to a 3D volumetric representation, and feed it into a 3D convolutional neural network(CNN) to directly produce the pose in 3D requiring no further process. Our system does not require the ground truth reference point for initialization, and our network architecture naturally integrates both local feature and global context in 3D space. To increase the coverage of the hand pose space of the training data, we render synthetic depth image by transferring hand pose from existing real image datasets. We evaluation our algorithm on two public benchmarks and achieve the state-ofthe-art performance. The synthetic hand pose dataset will be available. * indicates equal contribution. and optimized by a predefined hand model to reduce error that is significant in 3D space but not apparent in 2D image domain. Such post-process has been thoroughly studied but far from optimal, and [20] shows that even a directly searching of nearest neighbors pose in 3D space could also achieve performance close to the state of the art, which shows the benefits of directly solving pose estimation in 3D space.

Model-based Hand Pose Estimation for Generalized Hand Shape with Appearance Normalization

“…Although the NYU dataset annotates 36 joints, we use the same 14 joints for evaluation as most earlier works like [18] or [16]. 15.5 mm DeepModel [20] 16.9 mm DeepPrior [14] 19.8 mm DeepPrior++ [15] 12.3 mm Feedback [17] 16.2 mm Global to Local [13] 15.6 mm Hand3D [9] 17.6 mm HMDN [22] 16.3 mm Pose-REN [8] 11.8 mm REN [12] 12.7 mm SGN [22] 15.9 mm V2V-PoseNet [16] 8.4 mm the transformation parameters. Combining the appearance normalization pipeline with the Variable Hand CNN reduces the average joint location error by 3.3 mm.…”

“…Hand pose estimation approaches can be divided into three categories: 1) the generative, model-driven approaches that fit a hand model to the image observations by minimizing a cost function [4] [5] [6] [7], 2) the discriminative, datadriven approaches that directly predict the 3D joint locations from the images [8] [9] [10] [11] [12] [13] [14] [15] [16], and 3) the hybrid approaches that combine discriminative and generative elements [17] [18] [19] [20].…”

“…Discriminative methods play an important role because they are needed to initialize generative tracking methods and to recover in the case of tracking failure. State-of-theart discriminative methods use deep learning components such as 2D [8] [10] [12] [13] [14] [17] [15] [18] [21] [22] [20] or 3D [9] [11] [16] Convolutional Neural Networks (CNN) that might incorporate residual modules [8] [12] Fig. 1: Overview of our approach.…”

V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

“…Random forest-based methods [21,23,39,[41][42][43]48] provide fast and accurate performance. However, they utilize hand-crafted features and are overcome by recent CNN-based approaches [1,3,4,6,7,10,11,14,15,24,29,30,37,45,50,51] that can learn useful features by themselves. Tompson et al [45] firstly utilized CNN to localize hand keypoints by estimating 2D heatmaps for each hand joint.…”

Silhouette-Net: 3D Hand Pose Estimation from Silhouettes

“…Our binary-based approach is competitive with the state-of-the-art depth-based methods, and should be able to serve as a strong baseline on this task for future study. 3D hand pose estimation from depth maps: The performance of estimating 3D hand poses from depth maps has been improved rapidly (Choi et al 2015;Deng et al 2017;Ye, Yuan, and Kim 2016;Baek, In Kim, and Kim 2018;Wan et al 2018) in terms of prediction accuracy. The studies on depth-based hand pose estimation generally adopt either generative or discriminative methods.…”