Probabilistic Modeling for Human Mesh Recovery

Abstract

This paper focuses on the problem of 3D human reconstruction from 2D evidence. Although this is an inherently ambiguous problem, the majority of recent works avoid the uncertainty modeling and typically regress a single estimate for a given input. In contrast to that, in this work, we propose to embrace the reconstruction ambiguity and we recast the problem as learning a mapping from the input to a distribution of plausible 3D poses. Our approach is based on the normalizing flows model and offers a series of advantages. For conventional applications, where a single 3D estimate is required, our formulation allows for efficient mode computation. Using the mode leads to performance that is comparable with the state of the art among deterministic unimodal regression models. Simultaneously, since we have access to the likelihood of each sample, we demonstrate that our model is useful in a series of downstream tasks, where we leverage the probabilistic nature of the prediction as a tool for more accurate estimation. These tasks include reconstruction from multiple uncalibrated views, as well as human model fitting, where our model acts as a powerful image-based prior for mesh recovery. Our results validate the importance of probabilistic modeling, and indicate state-of-the-art performance across a variety of settings. Code and models are available at: https://www.seas.upenn.edu/ ˜nkolot/projects/prohmr.

Recovering 3D Human Mesh from Monocular Images: A Survey

“…Even though the occlusion has been extensively studied for years [20], [100], [102], [124], robustness and stability are still need to be improved. Besides, the visual evidence may be insufficient to identify a 3D reconstruction uniquely, recover several plausible reconstructions [125] or a pose distribution [126], [127] for one input is worthwhile.…”

“…Sengupta et al [103] assume simple multivariate Gaussian distributions over SMPL pose parameters θ and let the network to predict µ θ (I) and δ θ (I). ProHMR [126] models a conditional probability distribution p(θ|I) using Conditional Normalizing Flow, which is more powerful and expressive than Gaussian distributions. Sengupta et al [127] estimate a hierarchical matrix-Fisher distribution over the relative 3D rotation matrix of each joint.…”

“…The decoder in [122], [123] comprises of up-sampling and convolutional layers to generate UV position maps. In [126], a global feature vector is fed to a conditional normalizing flow to decode the probability distribution over pose parameters. [119], [120] leverage the transformer as a decoder to model nonlocal intersections among mesh vertices and joints, which complements convolutional operations.…”

“…Output Type a) Template paramters: [16], [17], [18], [19], [20], [21], [25], [37], [94], [97], [98], [99], [101], [102], [103], [104], [105], [106], [111], [112], [125], [129], [135], [140] b) 3D vertex coordinates: GraphCMR [117], Pose2Mesh [132], I2L-MeshNet [118], PC-HMR [121], METRO [120], Graphormer [119] c) Voxels: BodyNet [115], DeepHuman [116] d) UV position maps: DenseBody [122], DecoMR [123], Zhang et al [124] e) Probabilistic outputs: Biggs et al [125], Sengupta et al [127], [128], ProHMR [126] Intermediate/ Proxy Representation a) Silhouettes: Pavlakos et al [17], STRAPS [103], Skeleton2Mesh [98] b) Segmentations: NBF [18], Rueegg et al [129], STRAPS [103], Zanfir et al [105], HUND…”

“…The GHUM/GHUML models [81] rely on normalizing flows to represent skeleton kinematics. ProHMR [126] acts as an image-based pose prior to the fitting process, predicting the distribution of plausible poses given an input image. This distribution is modeled using conditional normalizing flows.…”