V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

Abstract

Most of the existing deep learning-based methods for 3D hand and human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of keypoints, such as hand or human body joints, via 2D convolutional neural networks (CNNs). The first weakness of this approach is the presence of perspective distortion in the 2D depth map. While the depth map is intrinsically 3D data, many previous methods treat depth maps as 2D images that can distort the shape of the actual object through projection from 3D to 2D space. This compels the network to perform perspective distortion-invariant estimation. The second weakness of the conventional approach is that directly regressing 3D coordinates from a 2D image is a highly nonlinear mapping, which causes difficulty in the learning procedure. To overcome these weaknesses, we firstly cast the 3D hand and human pose estimation problem from a single depth map into a voxel-to-voxel prediction that uses a 3D voxelized grid and estimates the per-voxel likelihood for each keypoint. We design our model as a 3D CNN that provides accurate estimates while running in real-time. Our system outperforms previous methods in almost all publicly available 3D hand and human pose estimation datasets and placed first in the HANDS 2017 frame-based 3D hand pose estimation challenge. The code is available in 1 .

Model-based Hand Pose Estimation for Generalized Hand Shape with Appearance Normalization

“…We use all three views (frontal + side views) of the images in the NYU training dataset for training. Although the NYU dataset annotates 36 joints, we use the same 14 joints for evaluation as most earlier works like [18] or [16]. 15.5 mm DeepModel [20] 16.9 mm DeepPrior [14] 19.8 mm DeepPrior++ [15] 12.3 mm Feedback [17] 16.2 mm Global to Local [13] 15.6 mm Hand3D [9] 17.6 mm HMDN [22] 16.3 mm Pose-REN [8] 11.8 mm REN [12] 12.7 mm SGN [22] 15.9 mm V2V-PoseNet [16] 8.4 mm the transformation parameters.…”

“…2 Our approach performs well, only being two to three millimeters worse in accuracy than the leading approaches. Compared to the best performing approach of mks0601 [16], our approach is conceptually and especially computationally much simpler. While the approach presented in [16] carries out inference with 3.5 frames per second on a single GPU, our approach achieves 838 frames per second and therefore has real-time capability.…”

“…Compared to the best performing approach of mks0601 [16], our approach is conceptually and especially computationally much simpler. While the approach presented in [16] carries out inference with 3.5 frames per second on a single GPU, our approach achieves 838 frames per second and therefore has real-time capability. Apart from that, our approach offers specific benefits: First, the kinematic layer offers access to the hand parameters, which might be useful for hand tracking applications.…”

“…Discriminative methods play an important role because they are needed to initialize generative tracking methods and to recover in the case of tracking failure. State-of-theart discriminative methods use deep learning components such as 2D [8] [10] [12] [13] [14] [17] [15] [18] [21] [22] [20] or 3D [9] [11] [16] Convolutional Neural Networks (CNN) that might incorporate residual modules [8] [12] Fig. 1: Overview of our approach.…”

“…These are fed into a kinematic layer that maps them to joint locations which are back-transformed into the initial coordinate system. [16] [15]. Relying on a large annotated training dataset, the discriminative methods either directly regress joint locations [8] [9] [11] [13] [14] [17] [15] [21] [22] [20] or output a probability density map for each joint [10] [16] [18].…”