gDNA: Towards Generative Detailed Neural Avatars

Chen, Xu; Jiang, Tianjian; Song, Jie; Yang, Jinlong; Black, Michael J.; Geiger, Andreas; Hilliges, Otmar

doi:10.1109/cvpr52688.2022.01978

Cited by 55 publications

(24 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When the parametric model is not available, these weights need to be discovered. To this end, recent works adopt learningbased solutions for discovering LBS weights [10,11,14,29,37,46,47,52]. They usually assume a shared canonical space and learn a canonical LBS weight field, which is used for deforming the body in the novel pose during inference.…”

Section: Related Workmentioning

confidence: 99%

Invertible Neural Skinning

Kant¹,

Siarohin²,

Guler³

et al. 2023

Preprint

View full text Add to dashboard Cite

Building animatable and editable models of clothed humans from raw 3D scans and poses is a challenging problem. Existing reposing methods suffer from the limited expressiveness of Linear Blend Skinning (LBS), require costly mesh extraction to generate each new pose, and typically do not preserve surface correspondences across different poses. In this work, we introduce Invertible Neural Skinning (INS) to address these shortcomings. To maintain correspondences, we propose a Pose-conditioned Invertible Network (PIN) architecture, which extends the LBS process by learning additional pose-varying deformations. Next, we combine PIN with a differentiable LBS module to build an expressive and end-to-end Invertible Neural Skinning (INS) pipeline. We demonstrate the strong performance of our method by outperforming the state-of-the-art reposing techniques on clothed humans and preserving surface correspondences, while being an order of magnitude faster. We also perform an ablation study, which shows the usefulness of our pose-conditioning formulation, and our qualitative results display that INS can rectify artefacts introduced by LBS well.

show abstract

Section: Related Workmentioning

confidence: 99%

Invertible Neural Skinning

Kant¹,

Siarohin²,

Guler³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…These methods require 3D ground-truth supervision, limiting their applicability to a few datasets and their ability to generalize beyond in-distribution poses. Recently, implicit representations have been used to learn a generative model of 3D people in clothing [13,6,17,50,14]. However, these approaches require ground truth posed 3D meshes or RGB-D video sequence to learn a model [72,76,40,3,2,1].…”

Section: Human Reconstruction Via Predictionmentioning

confidence: 99%

Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videos

Jena¹,

Chaudhari²,

Gee³

et al. 2023

Preprint

View full text Add to dashboard Cite

Human reconstruction and synthesis from monocular RGB videos is a challenging problem due to clothing, occlusion, texture discontinuities and sharpness, and framespecific pose changes. Many methods employ deferred rendering, NeRFs and implicit methods to represent clothed humans, on the premise that mesh-based representations cannot capture complex clothing and textures from RGB, silhouettes, and keypoints alone. We provide a counter viewpoint to this fundamental premise by optimizing a SMPL+D mesh and an efficient, multi-resolution texture representation using only RGB images, binary silhouettes and sparse 2D keypoints. Experimental results demonstrate that our approach is more capable of capturing geometric details compared to visual hull, mesh-based methods. We show competitive novel view synthesis and improvements in novel pose synthesis compared to NeRF-based methods, which introduce noticeable, unwanted artifacts. By restricting the solution space to the SMPL+D model combined with differentiable rendering, we obtain dramatic speedups in compute, training times (up to 24x) and inference times (up to 192x). Our method therefore can be used as is or as a fast initialization to NeRF-based methods.

show abstract

“…This is, in addition to pose and facial expression, the color depends on the last layer F of the geometry network and the normals n d in deformed space. This conditions the color prediction on the deformed geometry and local highfrequency details, which has been shown to be helpful [11,64]. Following [64], the normals are obtained via…”

Section: Lbs Regularizationmentioning

confidence: 99%