Identity-Aware Hand Mesh Estimation and Personalization from RGB Images

Kong, Deying; Zhang, Linguang; Chen, Liangjian; Ma, Haoyu; Yan, Xiaogang; Sun, Sumei; Liu, Xingwei; Han, Kun‐Yeun; Xie, Xiaohui

doi:10.1007/978-3-031-20065-6_31

Cited by 11 publications

(3 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, hand pose can be effectively estimated by images (Li, Gao, and Sang 2021) or point clouds (Cheng et al 2022;. In hand reconstruction, the parametric models like MANO (Romero, Tzionas, and Black 2017), NIMBLE (Li et al 2022b), have been used to recover the hand in the input image (Boukhayma, de Bem, and Torr 2019;Hasson et al 2019;Kong et al 2022;Cao et al 2021;Doosti et al 2020;Hasson et al 2020). In (Fan et al 2021;Ren, Zhu, and Zhang 2023;Kim, Kim, and Baek 2021;Zhang et al 2021;Li et al 2022a), the parametric models are employed to reconstruct two hands, where hand interactions and gestures could be simulated.…”

Section: Model-based Hand Reconstructionmentioning

confidence: 99%

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Gan,

Li,

Ren

et al. 2024

AAAI

View full text Add to dashboard Cite

Reconstructing high-fidelity hand models with intricate textures plays a crucial role in enhancing human-object interaction and advancing real-world applications. Despite the state-of-the-art methods excelling in texture generation and image rendering, they often face challenges in accurately capturing geometric details. Learning-based approaches usually offer better robustness and faster inference, which tend to produce smoother results and require substantial amounts of training data. To address these issues, we present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. Firstly, our approach predicts a parametric hand mesh model through Graph Convolutional Networks (GCN) based method from multi-view images. We further introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures, which is capable of preserving the mesh topology. In addition, we suggest an effective mesh-based neural rendering scheme to simultaneously generate photo-realistic image and optimize mesh geometry by fusing the pre-trained rendering network with vertex features. We conduct the comprehensive experiments on InterHand2.6M, DeepHandMesh and dataset collected by ourself, whose promising results show that our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality. Code and dataset are publicly available at https://github.com/agnJason/FMHR.

show abstract

Section: Model-based Hand Reconstructionmentioning

confidence: 99%

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Gan,

Li,

Ren

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Chen et al [15] propose a MANO-based [16] self-supervised hand shape reconstruction framework to simultaneously estimate the pose, shape, and texture of the hand and camera two-hand heatmap view. Methods [17,18,19] that investigate personal information to benefit hand pose and shape reconstruction have also been proposed.…”

Section: Single Hand Pose and Shape Estimation From Rgb Imagesmentioning

confidence: 99%

Interacting two-hand instance segmentation and pose estimation based on attention-induced separation

Sun,

Zhang

2024

Preprint

View full text Add to dashboard Cite

Estimating 3D hand poses from single RGB images is essential in real applications. However, the random use of single and two hands, the similarity and occlusion between hands, and the lack of depth clues make it challenging. To address this issue, we propose a framework for two-hand instance segmentation and pose estimation based on attention-induced separation. The framework first extracts hand joint heatmaps from images, which are then used as spatial attention to fuse with the input image along the channel dimension to implement hand instance segmentation. Subsequently, hand joint heatmaps and hand masks are combined to provide denser spatial attention and fuse with the input image along the channel dimension again for hand separation. Finally, this five-channel image is used for single-hand pose estimation. We extend the canonical 21-joint hand model to a 128-joint one to provide more effective hand-joint heatmap attention. Moreover, we utilize prior knowledge implied in the hand skeleton to help generate biomechanically feasible hand poses. Experimental results indicate that our framework outperforms state-of-the-art methods in the generalization ability of single-and two-hand pose estimation.

show abstract

“…[12] is a self-supervised 3D hand reconstruction network, which not only predicts 3D mesh but also outputs texture. [32] proposed an identity-aware hand mesh estimation model by regressing the parameters of MANO, which can incorporate the identity information to calibrate the shape parameters. [33] can find a balance between a parametric and non-parametric model to improve the accuracy of hand shape and pose.…”

Section: B Neural Parametric Hand Shape Reconstructionmentioning

confidence: 99%

PatientHandNet: 3-D Open-Palm Hand Reconstruction From Sparse Multiview Depth Images

Dai,

Zhao,

et al. 2023

IEEE Trans. Instrum. Meas.

View full text Add to dashboard Cite

Accurately reconstructing 3D hand shapes of patients is important for immobilization device customization, artificial limb generation, and hand disease diagnosis. Traditional 3D hand scanning requires multiple scans taken around the hand with a 3D scanning device. These methods require the patients to keep an open-palm posture during scanning, which is painful or even impossible for patients with impaired hand functions. Once multi-view partial point clouds are collected, expensive postprocessing is necessary to generate a high-fidelity hand shape. To address these limitations, we propose a novel deep-learning method dubbed PatientHandNet to reconstruct high-fidelity hand shapes in a canonical open-palm pose from multiple-depth images acquired with a single-depth camera. The hand poses in the depth images may vary, hand movements are allowed, facilitating the 3D scanning process in particular for patients with difficult conditions. The proposed method has strong operability since it is insensitive to the input pose, allowing for pose variations in the input depth images. We also proposed two novel datasets: a large-scale synthetic dataset to train our model and a realworld dataset with ground-truth hand biometrics extracted by an experienced anthropometrist. Extensive experimental results on the unseen synthetic data and real-world data demonstrate that the proposed method provides robust and easy-to-use hand shape reconstruction and outperforms the state-of-the-art methods in biometric accuracy terms.

show abstract

Identity-Aware Hand Mesh Estimation and Personalization from RGB Images

Cited by 11 publications

References 46 publications

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

Interacting two-hand instance segmentation and pose estimation based on attention-induced separation

PatientHandNet: 3-D Open-Palm Hand Reconstruction From Sparse Multiview Depth Images

Contact Info

Product

Resources

About