Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Abstract

Estimating 3D hand meshes from single RGB images is challenging, due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve the model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: inspired by recent work on human body, our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. HME demonstrates good performance for estimating diverse hand shapes and improves pose estimation accuracies. 2) Iterative testing refinement: Our fitting function is differentiable. We iteratively refine the initial estimate using the gradients, in the spirit of iterative model fitting methods like ICP. The idea is supported by the latest research on human body. 3) Self-data augmentation: collecting sized RGB-mesh (or segmentation mask)-skeleton triplets for training is a big hurdle. Once the model is successfully fitted to input RGB images, its meshes i.e. shapes and articulations, are realistic, and we augment view-points on top of estimated dense hand poses. Experiments using three RGB-based benchmarks show that our framework offers beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand shapes. Each technical component above meaningfully improves the accuracy in the ablation study.

A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods

“…Baek et al [119] proposed a model that is able to estimate the 3D skeleton structure of the hand from the RGB image and recover the hand shape from it. In their work, a 2D skeleton model was used to predict 21 joint points, and the 3D hand model used a generative mesh model named MANO [120] representing the hand grid based on 45-dimensional pose parameters and 10-dimensional shape parameters, which was used in some very recent work [121,122].…”

A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods

“…Baek et al [126] proposed a method that leverages the MANO hand model with 3D supervision, as well as hand segmentation masks and 2D supervision, to train a hand pose estimation CNN. Their network took an RGB image as input, estimated the 2D joint positions, and used them along with the image feature vector to predict the camera and the hand model parameters.…”

“…Iqbal [116] 0.710 AUC Dexter+Object [141] Multimodal Unimodal Inference Zhou et al [135] 0.948 AUC He et al [127] 12.40 mEPE Baek et al [126] 0.926 AUC Model-based Zhang et al [124] 0.901 AUC Iqbal et al [116] 13.41mEPE/0.940 AUC Theodoridis et al [118] 15.61 mEPE/0.907 AUC…”

“…Model-free Spurr et al [49] 19.73 mEPE/0.849 AUC Yang et al [133] 13.14 mEPE/0.943 AUC Ge et al [134] 0.920 AUC RHD [43] Multimodal Unimodal inference Cai et al [129] 0.887 AUC He et al [127] 3.96 mEPE Baek et al [126] 0.995 AUC Model-based Zhang et al [124] 0.995 AUC Theodoridis et al [118] 6.93 mEPE/0.997AUC Iqbal et al [116] 0.994 AUC…”

DeepFisheye

“…Hand pose estimation has been a major research area in the computer vision field, and it has benefited considerably from the rise of deep learning. Especially, hand pose estimation using a single color image as DeepFisheye, is an active re search area [3,47,63,72,78,82]. Zimmermann et al [81] suggested a deep learning pipeline that segments a hand in an image, identifies keypoints, and finally estimates the most likely hand pose.…”