HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation From a Single Depth Map

Abstract

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. The state-of-the-art methods directly regress 3D hand meshes from 2D depth images via 2D convolutional neural networks, which leads to artefacts in the estimations due to perspective distortions in the images.In contrast, we propose a novel architecture with 3D convolutions trained in a weakly-supervised manner. The input to our method is a 3D voxelized depth map, and we rely on two hand shape representations. The first one is the 3D voxelized grid of the shape which is accurate but does not preserve the mesh topology and the number of mesh vertices. The second representation is the 3D hand surface which is less accurate but does not suffer from the limitations of the first representation. We combine the advantages of these two representations by registering the hand surface to the voxelized hand shape. In the extensive experiments, the proposed approach improves over the state of the art by 47.8% on the SynHand5M dataset. Moreover, our augmentation policy for voxelized depth maps further enhances the accuracy of 3D hand pose estimation on real data. Our method produces visually more reasonable and realistic hand shapes on NYU and BigHand2.2M datasets compared to the existing approaches.

Graph-Based Hand-Object Meshes and Poses Reconstruction With Multi-Modal Input

“…In this section, we focus on the related works that reconstruct both hand and object from monocular input. We refer the reader to [11,12] for a detailed overview of works focusing on the reconstruction of hands and objects in isolation.…”

“…Content may change prior to final publication. [11,[38][39][40][41] utilized only depth maps instead of RGB images for estimating hand poses. However, many of the existing approaches focus on predicting the hand pose and aim to be stable in the presence of objects, but do not address the problem of simultaneous estimation of both hand and object.…”

“…The advantage of this representation is two-fold. First, depth map is inherently a 2.5D data which can be better represented in a 3D voxelized grid using binary quantization (occupancy grid) [11,38] or TSDF [51]. The TSDFbased representation is more effective than occupancy grid because TSDF allows to better encode the depth information by recognizing the voxels before and behind the observed surface [51].…”

3D Hand Gestures Segmentation and Optimized Classification Using Deep Learning

“…When considering any application for human-computer interaction like UAV control with hand gestures then it required accurate 3D hand pose estimation with key points and gestures recognition at the joint level which has various degrees of freedom (DoF) [1].…”

“…The analysis of 3D hand gestures used the most current methods for estimating 3D locations from monocular RGB images by understanding hand key points but unable to describe the 3D shape of a hand. In recent years, the pose estimation tasks have massive advancement and this can be accredited to key developments in the field of deep learning and a decrease in the cost of depth sensors [1]. However, the specified problem may exist to face many challenging factors.…”