Model-based 3D Hand Reconstruction via Self-Supervised Learning

Chen, Yujin; Tu, Zhigang; Kang, Di; Bao, Linchao; Zhang, Ying; Zhe, Xuefei; Chen, Ruizhi; Yuan, Junsong

doi:10.1109/cvpr46437.2021.01031

Cited by 86 publications

(36 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baek et al [52] address the challenging problem of hand-object interaction scenarios and combine a generative adversarial network and mesh renderer for guidance. Chen et al [3] employ an off-the-shell 2D pose detector [5] as a weaker 2D supervision, compared to human annotated 2D keypoints. By taking advantage of hand detector, the model can be trained on a wilder range of images without human label.…”

Section: Weakly-supervised Methodsmentioning

confidence: 99%

See 1 more Smart Citation

End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image

Ren¹,

Zhu²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we consider the challenging task of simultaneously locating and recovering multiple hands from single 2D image. Previous studies either focus on single hand reconstruction or solve this problem in a multi-stage way. Moreover, the conventional two-stage pipeline firstly detects hand areas, and then estimates 3D hand pose from each cropped patch. To reduce the computational redundancy in preprocessing and feature extraction, we propose a concise but efficient single-stage pipeline. Specifically, we design a multi-head autoencoder structure for multi-hand reconstruction, where each head network shares the same feature map and outputs the hand center, pose and texture, respectively. Besides, we adopt a weakly-supervised scheme to alleviate the burden of expensive 3D real-world data annotations. To this end, we propose a series of losses optimized by a stage-wise training scheme, where a multi-hand dataset with 2D annotations is generated based on the publicly available single hand datasets. In order to further improve the accuracy of the weakly supervised model, we adopt several feature consistency constraints in both single and multiple hand settings. Specifically, the keypoints of each hand estimated from local features should be consistent with the re-projected points predicted from global features. Extensive experiments on public benchmarks including FreiHAND, HO3D, InterHand2.6M and RHD demonstrate that our method outperforms the stateof-the-art model-based methods in both weakly-supervised and fully-supervised manners.

show abstract

Section: Weakly-supervised Methodsmentioning

confidence: 99%

“…R ECENTLY, a surge of research efforts [1] [2] [3] have been devoted to 3D hand reconstruction. In contrast to the conventional approaches relying on RGB-D sensor [4] or multiple view geometry [5], recovering 3D hand pose and its shape from single color image is more challenging due to the ambiguities in depth and scale.…”

Section: Introductionmentioning

confidence: 99%

End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image

Ren¹,

Zhu²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In recent years, scene understanding technologies for use in autonomous driving [19], virtual reality [20] and augmented reality [21] have developed rapidly. As the basic task of scene understanding, semantic segmentation technology based on pixel-by-pixel classification has been widely studied [22][23][24].…”

Section: Semantic Segmentation Based On Deep Learningmentioning

confidence: 99%

MAGNet: A camouflaged object detection network simulating the observation effect of a magnifier

Jiang

Cai

Jiang

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, protecting important objects by simulating animal camouflage has been widely used in many fields. Therefore, camouflaged object detection (COD) technology has emerged. COD is more difficult than traditional object detection techniques because of the high degree of fusion of camouflaged objects with the background. In this paper, we strive to more accurately identify camouflaged objects. Inspired by the use of magnifiers to search for hidden objects in pictures, we propose a COD network that simulates the observation effect of a magnifier, called the MAGnifier Network (MAGNet). Specifically, our MAGNet contains two parallel modules, i.e., the ergodic magnification module (EMM) and the attention focus module (AFM). The EMM is designed to mimic the process of a magnifier enlarging an image, and AFM is used to simulate the observation process in which human attention is highly focused on a region. The two sets of output camouflaged object maps are merged to simulate the observation of an object by a magnifier. In addition, a weighted key point area perception loss function, which is more applicable to COD, is designed based on two modules to give higher attention to the camouflaged object. Extensive experiments demonstrate that compared with 14 cutting-edge detection models, MAGNet can achieve the best comprehensive effect on eight evaluation metrics on the public COD dataset, and the segmentation accuracy is significantly improved. We also validate the models' generalization ability on a military camouflaged object dataset constructed in-house. Finally, we experimentally explore some extended applications of COD.

show abstract

“…Popular hand mesh estimation methods can be divided into five types, whose core ideas are based on the parametric model, voxel representation, implicit function, UV map, and vertex position, respectively. Model-based approaches [86,90,80,29,85,4,1,93,12,88,6,2,39,52,82,83] typically use MANO [65] as the parametric model, which factorizes a hand mesh into coefficients of shape and pose. This pipeline, however, is not suitable for usage with a lightweight network because the coefficient estimation is a highly abstract problem that ignores spatial correlations [22].…”

Section: Related Workmentioning

confidence: 99%

MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image

Chen¹,

Liu²,

Dong³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we propose a framework for singleview hand mesh reconstruction, which can simultaneously achieve high reconstruction accuracy, fast inference speed, and temporal coherence. Specifically, for 2D encoding, we propose lightweight yet effective stacked structures. Regarding 3D decoding, we provide an efficient graph operator, namely depth-separable spiral convolution. Moreover, we present a novel feature lifting module for bridging the gap between 2D and 3D representations. This module starts with a map-based position regression (MapReg) block to integrate the merits of both heatmap encoding and position regression paradigms to improve 2D accuracy and temporal coherence. Furthermore, MapReg is followed by pose pooling and pose-to-vertex lifting approaches, which transform 2D pose encodings to semantic features of 3D vertices. Overall, our hand reconstruction framework, called MobRecon, comprises affordable computational costs and miniature model size, which reaches a high inference speed of 83FPS on Apple A14 CPU. Extensive experiments on popular datasets such as FreiHAND, RHD, and HO3Dv2 demonstrate that our MobRecon achieves superior performance on reconstruction accuracy and temporal coherence. Our code is publicly available at https://github. com/SeanChenxy/HandMesh.

show abstract

Model-based 3D Hand Reconstruction via Self-Supervised Learning

Cited by 86 publications

References 32 publications

End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image

End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image

MAGNet: A camouflaged object detection network simulating the observation effect of a magnifier

MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image

Contact Info

Product

Resources

About