Relighting4D: Neural Relightable Human from Videos

Chen, Zhaoxi; Liu, Ziwei

doi:10.1007/978-3-031-19781-9_35

Cited by 25 publications

(6 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modeling surface materials and environment maps is already well‐explored for static scenes [ZXY*23, JLX*23b, ZSD*21, SDZ*21, BBJ*21], albeit the use of data priors is minimal, and most of the methods rely on a sufficiently large number of views to perform a per‐scene reconstruction. For non‐rigid scenes, methods that do intrinsic decomposition are based on human‐specific templates [ICN*23, CL22, ZYW*23, BLS*21, LMM*22, TAL*07, WSVT13, LWS*13]. A few early approaches exist for general non‐rigid scenes [WZN*14,GLD*19] but investigation with the new neural implicit representation paradigms is still missing.…”

Section: Remaining Challenges and Discussionmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real‐world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non‐rigidly moving scenes is a severely underconstrained and challenging problem. This state‐of‐the‐art report (STAR) offers the reader a comprehensive summary of state‐of‐the‐art techniques with monocular and multi‐view inputs such as data from RGB and RGB‐D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non‐rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state‐of‐the‐art techniques by reviewing recent approaches that use traditional and machine‐learning‐based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

show abstract

Section: Remaining Challenges and Discussionmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…We measure the MSE and SSIM metrics on the right, left, and two-hand sequences. The result shows that our method significantly outperforms the physically-based rendering baseline based on Relighting4D [6].…”

Section: Runtime Analysismentioning

confidence: 92%

“…Compared to eyes, hands exhibit significantly more diverse pose variations, making explicit visibility incorporation essential. Relight-ing4D [6] learns relightable materials of an articulated human under a single unknown illumination, but the fidelity of relighting is limited bu the expressiveness of their parametric BRDF model. In contrast to these methods, our approach enables relighting of articulate hand models that can be animated with a wide range of poses.…”

Section: Model-based Human Relightingmentioning

confidence: 99%

“…An alternative approach is to adopt a physicallybased BRDF model and render with the rendering equation. To validate the effectiveness of our neural rendering approach, we compare the teacher model against a physicallybased rendering baseline that is heavily inspired by the recent success of Relighting4D [6]. This physically-based baseline predicts material properties such as albedo, roughness, and a localized normal map per voxel, and feeds these into GGX microfacet BRDF [61] to obtain the RGB textures.…”

Section: B Comparison To Ggx Microfacet Brdf [61]mentioning

confidence: 99%

See 1 more Smart Citation

RelightableHands: Efficient Neural Relighting of Articulated Hand Models

Iwase¹,

Saito²,

Simon³

et al. 2023

Preprint

View full text Add to dashboard Cite

“…Invertible Neural Networks (INNs) (Dinh, Krueger, and Bengio 2015;Dinh, Sohl-Dickstein, and Bengio 2017;Behrmann et al 2019;Chen et al 2018;Kingma and Dhariwal 2018) are are capable of performing invertible transformations between the input and output space. They are widely used in generative models like Normalizing Flows (Kobyzev, Prince, and Brubaker 2020)…”

Section: Invertible Neural Networkmentioning

confidence: 99%

Relightable and Animatable Neural Avatars from Videos

Lin,

Zheng,

Yong

et al. 2024

AAAI

View full text Add to dashboard Cite

Lightweight creation of 3D digital avatars is a highly desirable but challenging task. With only sparse videos of a person under unknown illumination, we propose a method to create relightable and animatable neural avatars, which can be used to synthesize photorealistic images of humans under novel viewpoints, body poses, and lighting. The key challenge here is to disentangle the geometry, material of the clothed body, and lighting, which becomes more difficult due to the complex geometry and shadow changes caused by body motions. To solve this ill-posed problem, we propose novel techniques to better model the geometry and shadow changes. For geometry change modeling, we propose an invertible deformation field, which helps to solve the inverse skinning problem and leads to better geometry quality. To model the spatial and temporal varying shading cues, we propose a pose-aware part-wise light visibility network to estimate light occlusion. Extensive experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry and generates realistic shadows under different body poses. Code and data are available at https://wenbin-lin.github.io/RelightableAvatar-page.

show abstract

Relighting4D: Neural Relightable Human from Videos

Cited by 25 publications

References 57 publications

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

RelightableHands: Efficient Neural Relighting of Articulated Hand Models

Relightable and Animatable Neural Avatars from Videos

Contact Info

Product

Resources

About