Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning

Abstract

In this paper, we present a method that estimates reflectance and illumination information from a single image depicting a single-material specular object from a given class under natural illumination. We follow a data-driven, learning-based approach trained on a very large dataset, but in contrast to earlier work we do not assume one or more components (shape, reflectance, or illumination) to be known. We propose a two-step approach, where we first estimate the object's reflectance map, and then further decompose it into reflectance and illumination. For the first step, we introduce a Convolutional Neural Network (CNN) that directly predicts a reflectance map from the input image itself, as well as an indirect scheme that uses additional supervision, first estimating surface orientation and afterwards inferring the reflectance map using a learning-based sparse data interpolation technique. For the second step, we suggest a CNN architecture to reconstruct both Phong reflectance parameters and high-resolution spherical illumination maps from the reflectance map. We also propose new datasets to train these CNNs. We demonstrate the effectiveness of our approach for both steps by extensive quantitative and qualitative evaluation in both synthetic and real data as well as through numerous applications, that show improvements over the state-of-the-art.

Fast Spatially-Varying Indoor Lighting Estimation

“…More recently, Georgoulis et al [10] use deep learning to estimate lighting and reflectance from an object of known geometry, by first estimating its reflectance map (i.e., its "orientation-dependent" appearance) [30] and factoring it into lighting and material properties [9] afterwards.…”

All-Weather Deep Outdoor Lighting Estimation

“…Rematas et al [22] learn to infer a reflectance map (i.e., the convolution of incident illumination with surface reflectance) from a single image of an object. Subsequently, Georgoulis et al [6] factor reflectance maps into lighting and material properties [5]. Closely related to our work, Hold-Geoffroy et al [9] model outdoor lighting with the parametric Hošek-Wilkie sky model [10,11], and learn to estimate its parameters from a single image.…”

A Dataset of Multi-Illumination Images in the Wild

“…Several methods have explored this problem for outdoor images [28,14,15,19,31] as well as indoor environments [13]. Noting the lack of viable training data for indoor scenes, Gardner et al explicitly detect light sources in LDR panoramas [52].…”