Estimating Physically-Based Reflectance Parameters From a Single Image With GAN-Guided CNN

Abstract

We present a method that estimates the physically accurate reflectance of materials from a single image and reproduces real world materials which can be used in well-known graphics engines and tools. Recovering the BRDF (bidirectional reflectance distribution function) from a single image is an ill-posed problem due to the insufficient irradiance and geometry information as well as the insufficient samples on the BRDF parameters. The problem could be alleviated with a simplified representation of the surface reflectance such as Phong reflection model. Recent works have appealed that convolutional neural network successfully predicts parameters of empirical BRDF models for non-Lambertian surfaces. However, parameters of the physically-based model confront the problem of having non-orthogonal space, making it difficult to estimate physically meaningful results. In this paper, we propose a method to estimate parameters of a physically-based BRDF model from a single image. We focus on the metallic property of the physically-based model to enhance the estimation accuracy. Since metals and nonmetals have very different characteristics, our method processes them separately. Our method also generates auxiliary maps using a cGAN (conditional generative adversarial network) architecture to help in estimating more accurate BRDF parameters. Based on the experimental results, the auxiliary map is selected as an irradiance environment map for the metallic and a specular map for the nonmetallic. These auxiliary maps help to clarify the contributions of different actors, including light color, material color, specular component, and diffuse component, to the surface color. Our method first estimates whether the material on the input image is metallic or nonmetallic. Then, it estimates BRDF parameters using CNN (convolutional neural networks) architecture guided by generated auxiliary maps. Our results show that our method is effective to estimate BRDF parameters both on synthesized as well as real images.

Non-Lambertian surfaces and their challenges for visual SLAM

“…General equation for photometric stereo: Depth priors [24], [84] -Definition and collection of priors, computational complexity Masking methods [44], [45], [58] Uniform colour or limited colour variation Lack of robustness and generality, soft specularities Neural networks [9], [34], [68] -Lack of a well-defined BRDF, explainability and various special cases Hybrid models [22], [38], [53], [59] Surface reflection models…”

“…Another avenue involving neural networks has a different approach; they investigate how neural networks or machine learning could be used in tandem with physical models, bringing together the best of the two worlds: the accuracy and explainability of physical models and flexible and robust computations of neural networks. The first tidings of such approaches are already available, for example by Geourgoulis et al with a CNN learning the reflectance map of single non-Lambertian material and then fitting parameters to it [22], Li et al with a SVBRDF-modeling CNN [38], and Rhee and Lee with a GAN-guided CNN [53]. For a multi-spectral setting, thus accounting for wavelength as well, there is a study by Lv et al [42].…”