H4D: Human 4D Modeling by Learning Neural Compositional Representation

Jiang, Boyan; Zhang, Yinda; Wei, Xingkui; Xue, Xiangyang; Fu, Yanwei

doi:10.1109/cvpr52688.2022.01875

Cited by 17 publications

(5 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tang et al [TXJZ21] learn to autoencode object scan sequences with a spatio‐temporal autoencoder that predicts occupancy and correspondences. Similarly, Jiang et al [JZW*21] learn to disentangle object identity, object pose, and motion by formulating a neural ODE in the latent space. All these methods are trained on D‐FAUST.…”

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real‐world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non‐rigidly moving scenes is a severely underconstrained and challenging problem. This state‐of‐the‐art report (STAR) offers the reader a comprehensive summary of state‐of‐the‐art techniques with monocular and multi‐view inputs such as data from RGB and RGB‐D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non‐rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state‐of‐the‐art techniques by reviewing recent approaches that use traditional and machine‐learning‐based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

show abstract

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…In this category, point-based methods were also introduced [21,44,51,56,59] that learn the mapping between complete and partial shape point clouds with the benefit of being able to handle more general human shape topologies with low memory costs, using point based representations. With the aim to benefit from observations over time when available, STIF [58] and H4D [15] use spatio-temporal implicit functions or motion priors for human motion modeling. While accounting for the time dimension, the proposed methods do not model explicit temporal correspondences, such as the flow, hence only partially exploit the time dimension.…”

Section: Related Workmentioning

confidence: 99%

Human Body Shape Completion with Implicit Shape and Flow Learning

Zhou,

Meng,

Franco

et al. 2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…While their work shares the spirit of learning a motion manifold with ours, they learn a prior over fixed‐length sequences and operate on a set human skeletal topology. Likewise another 4D model specifically tied to the SMPL body model [JZW*22] uses gated recurrent units to model temporal dynamics of human shapes. To our knowledge, no generic disentangled 4D morphable shape model like ours exists for human faces.…”

Section: Related Workmentioning

confidence: 99%

Facial Animation with Disentangled Identity and Motion using Transformers

Chandran

Zoss

Groß

et al. 2022

Computer Graphics Forum

View full text Add to dashboard Cite

We propose a 3D+time framework for modeling dynamic sequences of 3D facial shapes, representing realistic non‐rigid motion during a performance. Our work extends neural 3D morphable models by learning a motion manifold using a transformer architecture. More specifically, we derive a novel transformer‐based autoencoder that can model and synthesize 3D geometry sequences of arbitrary length. This transformer naturally determines frame‐to‐frame correlations required to represent the motion manifold, via the internal self‐attention mechanism. Furthermore, our method disentangles the constant facial identity from the time‐varying facial expressions in a performance, using two separate codes to represent neutral identity and the performance itself within separate latent subspaces. Thus, the model represents identity‐agnostic performances that can be paired with an arbitrary new identity code and fed through our new identity‐modulated performance decoder; the result is a sequence of 3D meshes for the performance with the desired identity and temporal length. We demonstrate how our disentangled motion model has natural applications in performance synthesis, performance retargeting, key‐frame interpolation and completion of missing data, performance denoising and retiming, and other potential applications that include full 3D body modeling.

show abstract

H4D: Human 4D Modeling by Learning Neural Compositional Representation

Cited by 17 publications

References 40 publications

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Human Body Shape Completion with Implicit Shape and Flow Learning

Facial Animation with Disentangled Identity and Motion using Transformers

Contact Info

Product

Resources

About