3D Human Pose Machines with Self-supervised Learning

Abstract

Driven by recent computer vision and robotic applications, recovering 3D human poses has become increasingly important and attracted growing interests. In fact, completing this task is quite challenging due to the diverse appearances, viewpoints, occlusions and inherently geometric ambiguities inside monocular images. Most of the existing methods focus on designing some elaborate priors /constraints to directly regress 3D human poses based on the corresponding 2D human pose-aware features or 2D pose predictions. However, due to the insufficient 3D pose data for training and the domain gap between 2D space and 3D space, these methods have limited scalabilities for all practical scenarios (e.g., outdoor scene). Attempt to address this issue, this paper proposes a simple yet effective self-supervised correction mechanism to learn all intrinsic structures of human poses from abundant images. Specifically, the proposed mechanism involves two dual learning tasks, i.e., the 2D-to-3D pose transformation and 3D-to-2D pose projection, to serve as a bridge between 3D and 2D human poses in a type of "free" self-supervision for accurate 3D human pose estimation. The 2D-to-3D pose implies to sequentially regress intermediate 3D poses by transforming the pose representation from the 2D domain to the 3D domain under the sequence-dependent temporal context, while the 3D-to-2D pose projection contributes to refining the intermediate 3D poses by maintaining geometric consistency between the 2D projections of 3D poses and the estimated 2D poses. Therefore, these two dual learning tasks enable our model to adaptively learn from 3D human pose data and external large-scale 2D human pose data. We further apply our self-supervised correction mechanism to develop a 3D human pose machine, which jointly integrates the 2D spatial relationship, temporal smoothness of predictions and 3D geometric knowledge. Extensive evaluations on the Human3.6M and HumanEva-I benchmarks demonstrate the superior performance and efficiency of our framework over all the compared competing methods. Please find the code of this project at: http://www.sysu-hcp.net/3d pose ssl/ Index Terms-human pose estimation, convolutional neural networks, spatio-temporal modeling, self-supervised learning, geometric deep learning. ! K. Wang is with the

Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation

“…In order to train the network without explicit 3D pose annotations, the prior of camera projection geometry was commonly explored, and some geometry-driven methods were proposed. Among them, re-projection loss is one of the most widely used technique (Kanazawa et al 2018;Wu et al 2016;Wandt and Rosenhahn 2019;Pavllo et al 2019;Wang et al 2019;Brau and Jiang 2016). However, using re-projection loss alone cannot accurately con-Figure 2: The overall architecture of our method that follows a two-stage pipeline.…”

A Comprehensive Overview of Person Re-Identification Approaches

“…Self-supervised learning, more in line with the law of biological learning, has been very successful at Natural Language Processing (NLP). Recently, Wang et al [177] have used self-supervised learning to achieve accurate 3D human posture estimation.…”

Real‐time 3D human pose and motion reconstruction from monocular RGB videos

“…Learning‐based discriminative methods are quite popular for the real‐time 2D pose estimation of single or multiple characters . More recently, a number of methods have tackled a much harder problem, that of 3D skeletal estimation from single color images or videos . A common limitation of the discriminative methods in 3D pose reconstruction, though, is that they typically run off‐line, and because they do not take into consideration the temporal consistency of motion (they reconstruct the 3D joint positions on per image), the generated motion is oscillating.…”

“…The big challenge in this context is to learn rich features to encode depth, spatial and temporal relation of the body parts so as to ensure smooth motion reconstruction . While some recent methods run at high frame (e.g., other works), they are still unsuitable for use in closely interactive characters or crowds. This is because they require a tracked bounding box for each person, and thus, they can only reconstruct the motion of a single person at a time.…”