Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

Abstract

To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8× over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.

Deep learning tools for the measurement of animal behavior in neuroscience

“…3D point clouds or meshes ( Figure 1c), have become a popular and elegant way to capture the soft-tissue and shape of bodies, which are highly important features for person identification, fashion (i.e. clothing sales), and in medicine (21,42,43). However, state-of-the-art performance currently requires body-scanning of many subjects to make body models.…”

Resolving 3D Human Pose Ambiguities With 3D Scene Constraints

“…Recent methods based on deep learning, extend 3D human pose estimation to complex scenes [32,42,48,50] but the 3D accuracy is limited. To estimate human-scene interaction, however, more realistic body models are needed that include fully articulated hands such as in [31,49].…”

“…Thus, we penalize poses in which the body interpenetrates scene objects. We formulate this "exclusion principle" as a differentiable loss function that we incorporate into the SMPLify-X pose estimation method [49].…”

“…Our method extends SMPLify-X [49], which fits a 3D body model "top down" to "bottom up" features (e.g. 2D joint detections).…”

“…The former is easy to obtain today with many scanning technologies but, if the body model is not accurate, it does not make sense to reason about contact and inter-penetration. Consequently we use the SMPL-X body model [49], which is realistic enough to serve as a "proxy" for the real human in the 3D scene. In particular, the feet, hands, and body of the model have realistic shape and degrees of freedom.…”