Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation

Abstract

The neural network based approach for 3D human pose estimation from monocular images has attracted growing interest. However, annotating 3D poses is a labor-intensive and expensive process. In this paper, we propose a novel self-supervised approach to avoid the need of manual annotations. Different from existing weakly/self-supervised methods that require extra unpaired 3D ground-truth data to alleviate the depth ambiguity problem, our method trains the network only relying on geometric knowledge without any additional 3D pose annotations. The proposed method follows the two-stage pipeline: 2D pose estimation and 2D-to-3D pose lifting. We design the transform re-projection loss that is an effective way to explore multi-view consistency for training the 2D-to-3D lifting network. Besides, we adopt the confidences of 2D joints to integrate losses from different views to alleviate the influence of noises caused by the self-occlusion problem. Finally, we design a two-branch training architecture, which helps to preserve the scale information of re-projected 2D poses during training, resulting in accurate 3D pose predictions. We demonstrate the effectiveness of our method on two popular 3D human pose datasets, Human3.6M and MPI-INF-3DHP. The results show that our method significantly outperforms recent weakly/self-supervised approaches.

Towards Single 2D Image-Level Self-Supervision for 3D Human Pose and Shape Estimation

“…Kundu et al [64]'s method is the competitive self-supervised learning framework using the video priors; however, our method outperformed it even without any video priors. Li et al [54]'s method is the only baseline that outperforms our method and uses the self-supervised setting. However, the work is not directly comparable to ours, as its setting is easier as it estimates only skeletal joints and using multi-view priors.…”

“…Weakly or semi-supervised methods are designed to solve the issue of the lack of quality annotation by using the available easier annotations. In the context of the 3D human mesh estimation task, semisupervised learning uses 3D skeletons that are coarser than 3D meshes, while weaklysupervised learning uses either 2D annotation [36,51,52] or pseudo-3D annotation [53][54][55][56]. In [36], 2D keypoint annotations are exploited to estimate SMPL body model parameters from CNNs to recover human 3D meshes following predicted body part segmentation masks.…”

“…The authors of [52] suggest weakly-supervised solutions by adopting multi-view consistency loss in 2.5D pose representations leveraged by independent 2D annotated samples. In [53][54][55][56], multi-view geometry is used to resolve 3D depth ambiguity. We developed our original methods in the self-supervised setting only using the input 2D images; however, to compare with these weakly or semi-supervised approaches, we extended our model to be additionally learned by 2D or 3D skeletons if they are available.…”

3D Human Pose Estimation With Spatial Structure Information

“…Therefore, Wang et al [29] propose a novel stereo inspired neural network to generate high quality 3D pose labels for in-the-wild images. Li et al [30] and Chen et al [31] address this problem by training networks in self-supervision without annotated 3D data. Ramakrishna et al [32] propose an algorithm based on Projected Matching Pursuit.…”

Learning Temporal 3D Human Pose Estimation with Pseudo-Labels

“…Similar to the literature [29,17,19,5], the related work can be partitioned into supervised, unsupervised, self-and weakly-supervised learning. Below, we discuss the related approaches to our method.…”