Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locations

Abstract

The task of three-dimensional (3D) human pose estimation from a single image can be divided into two parts: (1) Two-dimensional (2D) human joint detection from the image and (2) estimating a 3D pose from the 2D joints. Herein, we focus on the second part, i.e., a 3D pose estimation from 2D joint locations. The problem with existing methods is that they require either (1) a 3D pose dataset or (2) 2D joint locations in consecutive frames taken from a video sequence. We aim to solve these problems. For the first time, we propose a method that learns a 3D human pose without any 3D datasets. Our method can predict a 3D pose from 2D joint locations in a single image. Our system is based on the generative adversarial networks, and the networks are trained in an unsupervised manner. Our primary idea is that, if the network can predict a 3D human pose correctly, the 3D pose that is projected onto a 2D plane should not collapse even if it is rotated perpendicularly. We evaluated the performance of our method using Human3.6M and the MPII dataset and showed that our network can predict a 3D pose well even if the 3D dataset is not available during training.

High Fidelity 3D Reconstructions with Limited Physical Views

“…Detected 2D GT 2D Iskakov et al [18] 20.8 -Remelli et al [43] 30.2 -Kadkhodamohammadi et al [21] 49.1 -Tome et al [49] 52.8 -Pavlakos et al [41] 56.9 -Multi-view Martinez [35] 57.0 -Rhodin et al [44] 51.6 -Kocabas et al [25] 45.04 -Kocabas et al (SS w/o R) [25] 70.67 -PRN [40] 124.5 86.4 RepNet [50] 65.1 38.2 Iqbal et al [17] 69.1 -Pose-GAN [29] 173.2 130.9 Deep NRSfM [27] -104.2 C3DPO [37] 153.0 95.6 MV NRSfM (Ours) 45.2 30.2…”

DOVE: Learning Deformable 3D Objects by Watching Videos

“…The idea is that the learned 3D model can be used to generate arbitrary views of an object, such that the discriminator network can provide supervisory signal to learn the geometry by telling whether the generated views are plausible or not. Examples include the works of [24,2], PlatonicGAN [16], HoloGAN [37], BlockGAN [38], Shelf-supervised learning [64] and GRAF [50]. This type of approach does not require information about specific viewpoints, but does require to know the distribution of viewpoints in the training data (as the discriminator is sensitive to that).…”

PAUL: Procrustean Autoencoder for Unsupervised Lifting

“…Instead of using classical NRSf M priors, recent works explore the use of deeper constraints. Generative Adversarial Networks (GANs) [14] are used to enforce realism of 2D reprojections across novel viewpoints [9,39,11,23]. These methods are only applicable for large datasets due to the requirement of learning GANs.…”

“…SH pts. [32] Pose-GAN [23] 130.9 173.2 C3DPO [32] 95.6 153.0 PRN [33] 86.4 124.5 PAUL 88.3 132.5 sion from 2 to 12 on different datasets. To account for the stochastic behavior due to network initialization and gradient descent on small datasets, we run the methods 10 times and visualize with average accuracy (solid lines) together with standard deviation (colored area).…”