Unite the People: Closing the Loop Between 3D and 2D Human Representations

Lassner, Christoph; Romero, Javier; Kiefel, Martin; Bogo, Federica; Black, Michael J.; Gehler, Peter V.

doi:10.1109/cvpr.2017.500

Cited by 552 publications

(534 citation statements)

References 44 publications

Supporting

Mentioning

531

Contrasting

Order By: Relevance

“…The work of [47] firstly built a synthetic dataset using graphics techniques, however, training only on this dataset is still difficult to produce a model that is applicable to real images. Lassner et al [20] proposed to apply the algorithm of [4] to obtain 3D body models for real images and then manually sift out the reasonable results, to build the final human body dataset. Unfortunately, the obtained 3D human shapes are still non-ideal and contain erroneous body part results.…”

Section: D Body Model Recoverymentioning

confidence: 99%

“…In this part, we evaluate the proposed human body recovery method of regressing the SMPL parameters on three public datasets i.e., SURREAL [47], UP-3D [20] and 3DPW [48]. Before the experimental studies, we first give an introduction to the datasets and related evaluation protocols.…”

Section: D Human Body Model Recoverymentioning

confidence: 99%

“…UP-3D. The details of this dataset are presented in [20]. To build it up, a large number of real images were collected, and then each of them was fitted by a SMPL body model.…”

Section: Datasets and Evaluation Protocolsmentioning

confidence: 99%

See 2 more Smart Citations

HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation

Zhou

Han

Jiang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

110

View full text Add to dashboard Cite

Estimating 3D human pose from a single image is a challenging task. This work attempts to address the uncertainty of lifting the detected 2D joints to the 3D space by introducing an intermediate state -Part-Centric Heatmap Triplets (HEMlets), which shortens the gap between the 2D observation and the 3D interpretation. The HEMlets utilize three joint-heatmaps to represent the relative depth information of the end-joints for each skeletal body part. In our approach, a Convolutional Network (ConvNet) is first trained to predict HEMlets from the input image, followed by a volumetric joint-heatmap regression. We leverage on the integral operation to extract the joint locations from the volumetric heatmaps, guaranteeing end-to-end learning. Despite the simplicity of the network design, the quantitative comparisons show a significant performance improvement over the best-of-grade methods (e.g. 20% on Human3.6M). The proposed method naturally supports training with "in-the-wild" images, where only weakly-annotated relative depth information of skeletal joints is available. This further improves the generalization ability of our model, as validated by qualitative comparisons on outdoor images. Leveraging the strength of the HEMlets pose estimation, we further design and append a shallow yet effective network module to regress the SMPL parameters of the body pose and shape. We term the entire HEMlets-based human pose and shape recovery pipeline HEMlets PoSh. Extensive quantitative and qualitative experiments on the existing human body recovery benchmarks justify the state-of-the-art results obtained with our HEMlets PoSh approach.

show abstract

Section: D Body Model Recoverymentioning

confidence: 99%

Section: D Human Body Model Recoverymentioning

confidence: 99%

See 1 more Smart Citation

HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation

Zhou

Han

Jiang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

110

View full text Add to dashboard Cite

show abstract

“…However, because they require each character to be tracked by a bounding box, they only reconstruct single‐person skeletons at a time, making them unsuitable for closely interacting characters. More recently, an enormous effort has been devoted to deep and convolutional methods that map all human pixels of an RGB image to 3D surface of the human body …”

Section: Related Workmentioning

confidence: 99%

“…More recently, an enormous effort has been devoted to deep and convolutional methods that map all human pixels of an RGB image to 3D surface of the human body. [45][46][47][48] Our method overcomes most of the prior work limitations, including the 3D capturing of multiple characters, the skeletal model constraints, and the production of smooth animation for the articulated character.…”

Section: Related Workmentioning

confidence: 99%

Real‐time 3D human pose and motion reconstruction from monocular RGB videos

Yiannakides

Aristidou

Chrysanthou

2019

Computer Animation & Virtual

View full text Add to dashboard Cite

Real‐time three‐dimensional (3D) pose estimation is of high interest in interactive applications, virtual reality, activity recognition, and most importantly, in the growing gaming industry. In this work, we present a method that captures and reconstructs the 3D skeletal pose and motion articulation of multiple characters using a monocular RGB camera. Our method deals with this challenging, but useful, task by taking advantage of the recent development in deep learning that allows two‐dimensional (2D) pose estimation of multiple characters and the increasing availability of motion capture data. We fit 2D estimated poses, extracted from a single camera via OpenPose, with a 2D multiview joint projections database that is associated with their 3D motion representations. We then retrieve the 3D body pose of the tracked character, ensuring throughout that the reconstructed movements are natural, satisfy the model constraints, are within a feasible set, and are temporally smooth without jitters. We demonstrate the performance of our method in several examples, including human locomotion, simultaneously capturing of multiple characters, and motion reconstruction from different camera views.

show abstract

Dense Pose Transfer

Neverova¹,

Güler

Kokkinos³

2018

Lecture Notes in Computer Science

207

157

View full text Add to dashboard Cite

Dense Pose Transfer 5 details is left to the supplemental material. We start by presenting the architecture of the predictive stream, and then turn to the surface-based stream, corresponding to the upper and lower rows of Fig. 1, respectively. Predictive streamThe predictive module is a conditional generative model that exploits the Dense-Pose system results for pose transfer. Existing conditional models indicate the target pose in the form of heat-maps from keypoint detectors [4], or part segmentations [3]. Here we condition on the concatenation of the input image and Dense-Pose results for the input and target images, resulting in an input of dimension 256×256×9. This provides conditioning that is both global (part-classification), and point-level (continuous coordinates), allowing the remaining network to exploit a richer source of information.The remaining architecture includes an encoder followed by a stack of residual blocks and a decoder at the end, along the lines of [28]. In more detail, this network comprises (a) a cascade of three convolutional layers that encode the 256×256×9 input into 64×64×256 activations, (b) a set of six residual blocks with 3×3×256×256 kernels, (c) a cascade of two deconvolutional and one convolutional layer that deliver an output of the same spatial resolution as the input. All intermediate convolutional layers have 3×3 filters and are followed by instance normalization [36] and ReLU activation. The last layer has tanh non-linearity and no normalization. Warping streamOur warping module performs pose transfer by performing explicit texture mapping between the input and the target image on the common surface UV-system. The core of this component is a Spatial Transformer Network (STN) [37] that warps according to DensePose the image observations to the UV-coordinate system of each surface part; we use a grid with 256×256 UV points for each of the 24 surface parts, and perform scattered interpolation to handle the continuous values of the regressed UV coordinates. The inverse mapping from UV to the output image space is performed by a second STN with a bilinear kernel. As shown in Fig. 3, a direct implementation of this module would often deliver poor results: the part of the surface that is visible on the source image is typically small, and can often be entirely non-overlapping with the part of the body that is visible on the target image. This is only exacerbated by DensePose failures or systematic errors around the part seams. These problems motivate the use of an inpainting network within the warping module, as detailed below.Inpainting autoencoder. This model allows us to extrapolate the body appearance from the surface nodes populated by the STN to the remainder of the surface. Our setup requires a different approach to the one of other deep inpainting methods [33], because we never observe the full surface texture during training. We handle the partially-observed nature of our training signal by using

show abstract

Unite the People: Closing the Loop Between 3D and 2D Human Representations

Cited by 552 publications

References 44 publications

HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation

HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation

Real‐time 3D human pose and motion reconstruction from monocular RGB videos

Dense Pose Transfer

Contact Info

Product

Resources

About