Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Abstract

Recent studies have shown remarkable advances in 3D human pose estimation from monocular images, with the help of large-scale in-door 3D datasets and sophisticated network architectures. However, the generalizability to different environments remains an elusive goal.In this work, we propose a geometry-aware 3D representation for the human pose to address this limitation by using multiple views in a simple auto-encoder model at the training stage and only 2D keypoint information as supervision. A view synthesis framework is proposed to learn the shared 3D representation between viewpoints with synthesizing the human pose from one viewpoint to the other one. Instead of performing a direct transfer in the raw imagelevel, we propose a skeleton-based encoder-decoder mechanism to distil only pose-related representation in the latent space. A learning-based representation consistency constraint is further introduced to facilitate the robustness of latent 3D representation. Since the learnt representation encodes 3D geometry information, mapping it to 3D pose will be much easier than conventional frameworks that use an image or 2D coordinates as the input of 3D pose estimator. We demonstrate our approach on the task of 3D human pose estimation. Comprehensive experiments on three popular benchmarks show that our model can significantly improve the performance of state-of-the-art methods with simply injecting the representation as a robust 3D prior.

Automatic 3D Landmark Extraction System Based on an Encoder–Decoder Using Fusion of Vision and LiDAR

“…3D poses are obtained by converting 2D poses to 3D one through epipolar geometry [10]. The 2D pose is converted into a 3D pose by a trained deep learning model [11]. A study evaluated whether the 3D poses of the user's motions obtained by the deep learning model are actually representative of the user's motions, and thus, confirmed that useful 3D poses are provided [19].…”

“…The deep learning model analyzes RGB images and provides 3D poses of the user's motions [10,11]. A 2D pose is calculated by a pre-trained deep learning model using the RGB image.…”

Semi-Supervised 3D Human Pose Estimation by Jointly Considering Temporal and Multiview Information

“…In addition to using temporal information, if the action was captured from multiple views, information from different views may be complementary. Chen et al [6] extracted features from two different views, and transformed the features from one view into another. If features from different views are transformed well, these features can be used to estimate better 3D human pose.…”

“…Pavllo et al [17] proposed a temporal convolution model to take 2D skeleton sequences as input to estimate 3D human poses. To utilize rich information from the spatial domain, Chen et al [6] proposed to jointly consider videos captured from two different views. 2D skeletons are first detected from each view.…”

“…We integrate the ideas proposed in [17] and [6], and make significant modifications to join temporal information and multiview information into a unified semi-supervised framework, which is trained based on adversarial learning. Although each separate module has been proposed before, how performance can be improved by jointly taking them in a unified model was not investigated before, and this is the main contribution of this paper.…”