Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

Abstract

Model-based human pose estimation is currently approached through two different paradigms. Optimizationbased methods fit a parametric body model to 2D observations in an iterative manner, leading to accurate imagemodel alignments, but are often slow and sensitive to the initialization. In contrast, regression-based methods, that use a deep network to directly estimate the model parameters from pixels, tend to provide reasonable, but not pixel accurate, results while requiring huge amounts of supervision. In this work, instead of investigating which approach is better, our key insight is that the two paradigms can form a strong collaboration. A reasonable, directly regressed estimate from the network can initialize the iterative optimization making the fitting faster and more accurate. Similarly, a pixel accurate fit from iterative optimization can act as strong supervision for the network. This is the core of our proposed approach SPIN (SMPL oPtimization IN the loop). The deep network initializes an iterative optimization routine that fits the body model to 2D joints within the training loop, and the fitted estimate is subsequently used to supervise the network. Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network. We demonstrate the effectiveness of our approach in different settings, where 3D ground truth is scarce, or not available, and we consistently outperform the state-of-the-art model-based pose estimation approaches by significant margins. The project website with videos, results, and code can be found at https://seas.upenn.edu/˜nkolot/projects/spin.

HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation

“…12. Here, HMR [16] and SPIN [18] are selected as two representative body mesh recovery approaches. Given an input image, the output body mesh of each method is shown in two views.…”

“…Qualitative comparisons of our method with some existing ones on human body model recovery. For each example, the input image is first shown, which is followed by the results of HMR [16], SPIN [18] and ours. For each resulting body mesh, two views are provided for visualization.…”

“…For example, the strategy of adversarial learning was utilized in [16], where a discriminator is exploited to constrain the regressed parameters against a reasonable distribution. Given the regressed SMPL models, Kolotouros et al [18] further adopted the approach of [4] to obtain better parameters to fit the input images. They then directly set an extra objective for the regression model to enforce its output parameters equal to the optimized ones.…”

“…We also report the accuracy of the output 3D pose, where the average per-joint Euclidean distance between the estimated pose (with the hip joint aligned) and the groundtruth is used (which is referred to as "joint"). For the dataset of 3DPW, we follow the works of [16], [18] to evaluate the reconstruction error of 3D poses, which is noted as "Rec. Error".…”

Towards Contactless Patient Positioning

“…The θ and β estimated by an RDF model trained with the learning objective in Equation 7 can be further fine-tuned in an online fashion. Specifically, given a good initialization and a sufficient number of iterations, the work of Kolotouros et al [19] noted that using an optimization-based iterative approach (e.g., SMPLify [20]) to fit body keypoints (from detection in Fig. 2) typically leads to better results than regressionbased approaches.…”