Model-based Hand Pose Estimation for Generalized Hand Shape with Appearance Normalization

Abstract

Since the emergence of large annotated datasets, state-of-the-art hand pose estimation methods have been mostly based on discriminative learning. Recently, a hybrid approach has embedded a kinematic layer into the deep learning structure in such a way that the pose estimates obey the physical constraints of human hand kinematics. However, the existing approach relies on a single person's hand shape parameters, which are fixed constants. Therefore, the existing hybrid method has problems to generalize to new, unseen hands. In this work, we extend the kinematic layer to make the hand shape parameters learnable. In this way, the learnt network can generalize towards arbitrary hand shapes. Furthermore, inspired by the idea of Spatial Transformer Networks, we apply a cascade of appearance normalization networks to decrease the variance in the input data. The input images are shifted, rotated, and globally scaled to a similar appearance. The effectiveness and limitations of our proposed approach are extensively evaluated on the Hands 2017 challenge dataset and the NYU dataset.

Disentangling Latent Hands for Image Synthesis and Pose Estimation

“…Much of the progress made in hand pose estimation have focused on using depth image inputs [5,6,7,8,11,14,15,18,19,32,33,35]. State-of-the-art methods use a convolutional neural network (CNN) architecture, with the majority of works treating the depth input as 2D pixels, though a few more recent approaches treat depth inputs as a set of 3D points and or voxels [7,5,15].…”

Multi-view Image-based Hand Geometry Refinement using Differentiable Monte Carlo Ray Tracing

“…Stereo input was similarly explored [24,40,47]. With the wide availability of commodity depth sensors around 2010, the research also focused on monocular depth or RGBD input [10,20,29,31,36,37,43,55,56,57,61]. However, shortly thereafter, the success of deep learning lead to the proliferation of robust systems that perform on monocular RGB input [3,4,9,11,13,17,22,34,41,48,53,54,58,62,63].…”

“…The accuracy achieved with the availability of depth information yielded the first practical real-time systems [37], and enabled on-the-fly adjustment of the bone lengths to the observed hand [29,59]. The development and availability of a parametric hand pose and shape model [49] in turn fueled research towards estimating the surface of the observed hand [2,11,22,32,61]. Finally, some very recent works are presenting the first attempts to model the appearance of the hand to some extent [5,32,44].…”