Probabilistic 3D Human Shape and Pose Estimation from Multiple Unconstrained Images in the Wild

Abstract

This paper addresses the problem of 3D human body shape and pose estimation from RGB images. Recent progress in this field has focused on single images, video or multi-view images as inputs. In contrast, we propose a new task: shape and pose estimation from a group of multiple images of a human subject, without constraints on subject pose, camera viewpoint or background conditions between images in the group. Our solution to this task predicts distributions over SMPL body shape and pose parameters conditioned on the input images in the group. We probabilistically combine predicted body shape distributions from each image to obtain a final multi-image shape prediction. We show that the additional body shape information present in multi-image input groups improves 3D human shape estimation metrics compared to single-image inputs on the SSP-3D dataset and a private dataset of tape-measured humans. In addition, predicting distributions over 3D bodies allows us to quantify pose prediction uncertainty, which is useful when faced with challenging input images with significant occlusion. Our method demonstrates meaningful pose uncertainty on the 3DPW dataset and is competitive with the state-of-the-art in terms of pose estimation metrics.

Probabilistic Estimation of 3D Human Shape and Pose with a Semantic Local Parametric Model

“…Learning-based methods yield impressive 3D pose estimates in-the-wild, but shape predictions are often inaccurate, due to the lack of shape diversity in training datasets. Some recent works improve shape estimates using synthetic training data [37,38,39,40], which we adopt in our method.…”

“…However, body shape estimates tend to be inaccurate or inconsistent for subjects in-the-wild. Recently, [38,39] attempt to predict accurate and consistent body shapes from multiple images of a subject, without assuming a fixed body pose or background and lighting conditions. This involves (i) predicting independent Gaussian distributions (i.e.…”

“…To this end, we extend [39] by predicting distributions over local semantic body shape measurements (e.g. chest width, arm length, calf circumference, etc), conditioned on an input image.…”

“…Second, a deep neural network predicts means and variances of Gaussian distributions over SMPL pose parameters and body measurements, conditioned on the input proxy representations. Third, body measurements from each image are probabilistically combined [39] to give a final measurements estimate, which is converted into a full body shape estimate using our measurements-to-β s regressor and the SMPL function [26]. Probabilistic combination intuitively amounts to uncertainty-weighted averaging (Equation 5) -since our measurement distributions are able to better capture local shape uncertainty than independent Gaussians over SMPL β s [39], we obtain improved body shape estimation accuracy.…”

“…Third, body measurements from each image are probabilistically combined [39] to give a final measurements estimate, which is converted into a full body shape estimate using our measurements-to-β s regressor and the SMPL function [26]. Probabilistic combination intuitively amounts to uncertainty-weighted averaging (Equation 5) -since our measurement distributions are able to better capture local shape uncertainty than independent Gaussians over SMPL β s [39], we obtain improved body shape estimation accuracy. This is quantitatively corroborated by shape metrics on the SSP-3D dataset [37], as well two private datasets of tape-measured humans, in an A-pose and in varying poses.…”