3D Human Shape Reconstruction from a Polarization Image

“…There are approaches for specialized problems, such as jointly identifying a person and the object they are interacting with [27], or specifying on low-resolution images [25]. Camera systems with alternative capturing types, like polarized 2D images [30], images with a depth channel [31], or images from an event-based camera [24] can be used to improve different aspects of body model parameter fitting based on individual images.…”

Section: Fitting a Body Model To Imagesmentioning

confidence: 99%

Imposing Temporal Consistency on Deep Monocular Body Shape and Pose Estimation

Zimmer¹,

Hilsmann²,

Morgenstern³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accurate and temporally consistent modeling of human bodies is essential for a wide range of applications, including character animation, understanding human social behavior and AR/VR interfaces. Capturing human motion accurately from a monocular image sequence is still challenging and the modeling quality is strongly influenced by the temporal consistency of the captured body motion. Our work presents an elegant solution for the integration of temporal constraints in the fitting process. This does not only increase temporal consistency but also robustness during the optimization. In detail, we derive parameters of a sequence of body models, representing shape and motion of a person, including jaw poses, facial expressions, and finger poses. We optimize these parameters over the complete image sequence, fitting one consistent body shape while imposing temporal consistency on the body motion, assuming linear body joint trajectories over a short time. Our approach enables the derivation of realistic 3D body models from image sequences, including facial expression and articulated hands. In extensive experiments, we show that our approach results in accurately estimated body shape and motion, also for challenging movements and poses. Further, we apply it to the special application of sign language analysis, where accurate and temporal consistent motion modelling is essential, and show that the approach is well-suited for this kind of application.

show abstract

“…Similarly, we use this data for state-of-the-art comparison. HumanAct12 is adapted from the PHSPD dataset [66] that releases SMPL pose parameters and root translations in camera coordinates for 1191 videos. HumanAct12 temporally trims the videos, annotates them into 12 action categories, and only provides their joint coordinates in a canonical frame.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

Petrovich¹,

Black²,

Varol³

2021

Preprint

View full text Add to dashboard Cite

We tackle the problem of action-conditioned generation of realistic and diverse human motion sequences. In contrast to methods that complete, or extend, motion sequences, this task does not require an initial pose or sequence. Here we learn an action-aware latent representation for human motions by training a generative variational autoencoder (VAE). By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action. Specifically, we design a Transformer-based architecture, ACTOR, for encoding and decoding a sequence of parametric SMPL human body models estimated from action recognition datasets. We evaluate our approach on the NTU RGB+D, HumanAct12 and UESTC datasets and show improvements over the state of the art. Furthermore, we present two use cases: improving action recognition through adding our synthesized data to training, and motion denoising. Our code and models will be made available.

show abstract

3D Human Shape Reconstruction from a Polarization Image

Cited by 39 publications

References 62 publications

Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction

Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction

Imposing Temporal Consistency on Deep Monocular Body Shape and Pose Estimation

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

Contact Info

Product

Resources

About