Efficient reflectance capture using an autoencoder

Abstract

We propose a novel framework that automatically learns the lighting patterns for efficient reflectance acquisition, as well as how to faithfully reconstruct spatially varying anisotropic BRDFs and local frames from measurements under such patterns. The core of our framework is an asymmetric deep autoencoder, consisting of a nonnegative, linear encoder which directly corresponds to the lighting patterns used in physical acquisition, and a stacked, nonlinear decoder which computationally recovers the BRDF information from captured photographs. The autoencoder is trained with a large amount of synthetic reflectance data, and can adapt to various factors, including the geometry of the setup and the properties of appearance. We demonstrate the effectiveness of our framework on a wide range of physical materials, using as few as 16 ~ 32 lighting patterns, which correspond to 12 ~ 25 seconds of acquisition time. We also validate our results with the ground truth data and captured photographs. Our framework is useful for increasing the efficiency in both novel and existing acquisition setups.

Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images

“…Optimizing with Auto-encoders. Kang et al [2018] learn illumination patterns for efficient capture of surface reflectance properties. Key to their method is an asymmetric auto-encoder that that features a linear non-negative encoder that corresponds to the acquisition lighting, and a non-linear decoder that maps the measurements to reflectance information, which in turn is fitted to an analytical BRDF model.…”

Light stage super-resolution

“…Recently, many approaches for acquiring a sparse light transport matrix have been developed, including methods based on compressive sensing [Peers et al 2009;Sen and Darabi 2009], kernel Nystrom [Wang et al 2009], optical computing [O'Toole and Kutulakos 2010] and neural networks [Ren et al 2013Kang et al 2018]. However, these methods are not designed for the light stage and are largely orthogonal to our approach.…”

Learned Fitting of Spatially Varying BRDFs

“…In a quite different line of works Aittala et al [AWL13] and Fichet et al [FSH16] exploit the frequency domain to obtain reliably reflectance estimates from few samples. Similar to the approach of Aittala et al which uses frequency domain illumination patterns, Kang et al [KCW∗18] use an autoencoder to learn arbitrary illumination patterns, allowing them to reduce acquisition times to a few seconds. Ren et al [RWS∗11], Riviere et al [RPG16] and Albert et al [ACGO18] avoid the complexity and acquisition effort of calibrated setups and instead estimate SVBRDFs from mobile phone video.…”

“…Learning based methods have been previously applied in the field of appearance capture. In the last years a series of works that utilize deep learning models to extract reflectance from varying kinds of input images has been published [AAL16; LDPT17; YLD∗18; KCW∗18; LSC18; LXR∗18; DAD∗18; VCGL19]. The trend is to feed very sparsely measured inputs into a network, usually one or a few photographs, to obtain an estimate of the observed surface reflectance.…”