Deep reflectance fields

Meka, Abhimitra; Häne, Christian; Pandey, Rohit; Zollhöfer, Michael; Fanello, Sean; Fyffe, Graham; Kowdle, Adarsh; Yu, Xing; Busch, Jay; Dourgarian, Jason; Denny, Peter; Bouaziz, Samir; Lincoln, Peter; Whalen, Matt; Harvey, Geoff; Taylor, Jonathan; Izadi, Shahram; Tagliasacchi, Andrea; Debevec, Paul; Theobalt, Christian; Valentin, Julien; Rhemann, Christoph

doi:10.1145/3306346.3323027

Cited by 75 publications

(24 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They however do not estimate any specular reflectance. Most recently, Meka et al [2019] have proposed efficient dynamic performance relighting using capture with RGB-multiplexed unpolarized spherical gradient illumination and complement pairs which are then used as input to a convolutional deep network to predict relit facial performance under a novel lighting. This however requires a database of acquired facial reflectance fields in multiple expressions to train the deep network, and does not result in reflectance maps or geometry that can be used in a standard rendering pipeline.…”

Section: Related Workmentioning

confidence: 99%

Single-shot high-quality facial geometry and skin appearance capture

et al. 2020

View full text Add to dashboard Cite

We propose a new light-weight face capture system capable of reconstructing both high-quality geometry and detailed appearance maps from a single exposure. Unlike currently employed appearance acquisition systems, the proposed technology does not require active illumination and hence can readily be integrated with passive photogrammetry solutions. These solutions are in widespread use for 3D scanning humans as they can be assembled from off-the-shelf hardware components, but lack the capability of estimating appearance. This paper proposes a solution to overcome this limitation, by adding appearance capture to photogrammetry systems. The only additional hardware requirement to these solutions is that a subset of the cameras are cross-polarized with respect to the illumination, and the remaining cameras are parallel-polarized. The proposed algorithm leverages the images with the two different polarization states to reconstruct the geometry and to recover appearance properties. We do so by means of an inverse rendering framework, which solves per texel diffuse albedo, specular intensity, and high-resolution normals, as well as global specular roughness considering the subsurface scattering nature of skin. We show results for a variety of human subjects of different ages and skin typology, illustrating how the captured fine-detail skin surface and subsurface scattering effects lead to realistic renderings of their digital doubles, also in different illumination conditions.

show abstract

Section: Related Workmentioning

confidence: 99%

Single-shot high-quality facial geometry and skin appearance capture

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Relighting approaches modify the incident illumination on the face [Meka et al 2019;Peers et al 2007;Shu et al 2017;Sun et al 2019;Zhou et al 2019]. Earlier works [Peers et al 2007;Shu et al 2017] require an exemplar portrait image that has been taken under the target illumination conditions.…”

Section: Portrait Relightingmentioning

confidence: 99%

“…Earlier works [Peers et al 2007;Shu et al 2017] require an exemplar portrait image that has been taken under the target illumination conditions. More recent techniques use deep generative models [Meka et al 2019;Sun et al 2019;Zhou et al 2019] and can relight images based on an environment map. Zhou et al [2019] train a relighting technique based on a large corpus of synthetic images.…”

Section: Portrait Relightingmentioning

confidence: 99%

“…While training with light stage data leads to high-quality results, their scarcity and careful recording protocol can limit their adaptation. Meka et al [2019] showed that the 4D reflectance field can be estimated from two color gradient images captured in a light stage. This provides more movement flexibility for the subject during recording, and hence takes an important step towards capturing relightable video.…”

Section: Portrait Relightingmentioning

confidence: 99%

“…Some model-based methods to realistically edit human expression [Thies et al 2019[Thies et al , 2016 and head pose fundamentally require video input and do not work on single images. Other editing approaches are image-based and cannot be controlled by intuitive parametric controls [Averbuch-Elor et al 2017;Geng et al 2018;Siarohin et al 2019;Wang et al 2019a;Zakharov et al 2019], only enable editing of a single semantic parameter dimension, e.g., scene illumination [Meka et al 2019;Sun et al 2019;Zhou et al 2019], or do not photo-realistically synthesize some important features such as hair [Nagano et al 2018].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Pie

et al. 2020

Self Cite

View full text Add to dashboard Cite

portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image. Semantic editing in parameter space is achieved based on StyleRig, a pretrained neural network that maps the control space of a 3D morphable face model to the latent space of the GAN. We design a novel hierarchical non-linear optimization problem to obtain the embedding. An identity preservation energy term allows spatially coherent edits while maintaining facial integrity. Our approach runs at interactive frame rates and thus allows the user to explore the space of possible edits. We evaluate our approach on a wide set of portrait photos, compare it to the current state of the art, and validate the effectiveness of its components in an ablation study. CCS Concepts: • Computing methodologies Image manipulation.

show abstract

A multiresolution network architecture for deferred neural lighting

Ren

et al. 2022

Computer Animation & Virtual

View full text Add to dashboard Cite

We present a novel multiresolution network architecture for deferred neural lighting. The key idea is to explicitly separate the processing of appearance at different spatial resolutions, leading to considerably improved high-frequency details as well as temporal stability in animation sequences with varying view conditions. Moreover, our network is only half the size of the original one, and requires less training data to converge to satisfactory results. The network is tested over five captured datasets from deferred neural lighting and may be extended to other neural appearance techniques, such as NeRF or neural textures.

show abstract

Deep reflectance fields

Cited by 75 publications

References 46 publications

Single-shot high-quality facial geometry and skin appearance capture

Single-shot high-quality facial geometry and skin appearance capture

Pie

A multiresolution network architecture for deferred neural lighting

Contact Info

Product

Resources

About