Capturing Implicit Spatial Cues for Monocular 3d Hand Reconstruction

Abstract

With the development of the parameterized hand model (e.g. MANO), it is possible to reconstruct the 3D hand mesh from a single 2D hand image by learning a few hand model parameters, rather than estimating hundreds of vertices on the mesh. However, it is highly non-linear to learn these parameters from the 2D hand image, as there is no explicit spatial correspondence between these parameters and image pixels. In this paper, we successfully leverage the graph convolutional network (GCN) to capture implicit spatial cues for fitting the well-known MANO hand model, thus greatly improving the performance of monocular 3D hand reconstruction. Our proposed MANO-GCN establishes the spatial hand mesh and hand joints graph for learning MANO parameters, with node features propagated along edges to utilize the spatial information. Among all monocular 3D hand reconstruction methods with MANO hand model, MANO-GCN achieves stateof-the-art accuracy on public FreiHAND and HO-3D benchmarks, without any bells and whistles. Code is available at https://github.com/ChenJoya/manogcn.

Adaptive graph residual network for hand shape estimation in single images

“…Our method outperforms the 10 recent baselines under all the six evaluation metrics. The superior performance of our method and MANO-GCN [24] over the CNN-based baselines demonstrates that a GCN is suitable for modeling both sparse hand joints and dense mesh vertices. MANO-GCN also adopts a two-branch architecture, but it is inferior to our method under all the six evaluation metrics.…”

“…Hand shapes were regressed from the latent features by a Chebyshev spectral GCN. Q. Wu et al [24] extracted latent features by HRNet [20]. Hand poses and shapes were estimated by combining a differentiable MANO layer and a 3-layer Chebyshev spectral GCN.…”

“…Based on the adaptive graph convolution, we design a 9-layer graph residual network with a double-skip-connection architecture, as shown in Figure 1. Following [24,39,40], HRNet [41] is used as the backbone to extract high-resolution spatial features. The latent features extracted at the first branch are fed into a fully-connected layer to obtain the pose vector θ and the shape vector β.…”

“…J 0 and V 0 are concatenated with the pose features and the shape features to generate PoseGCN features and MeshGCN features, respectively. Different from the existing GCN-based methods [10,12,24] that adopt a simple multilayer architecture, we introduce a double-skip connection to our graph residual network. The double-skip connection is added to the input and output of PoseGCN and MeshGCN at the 2-th, 4-th, 6-th, and 8-th layers.…”