Highlight-aware two-stream network for single-image SVBRDF acquisition

Abstract

This paper addresses the task of estimating spatially-varying reflectance (i.e., SVBRDF) from a single, casually captured image. Central to our method is a highlight-aware (HA) convolution operation and a two-stream neural network equipped with proper training losses. Our HA convolution, as a novel variant of standard (ST) convolution, directly modulates convolution kernels under the guidance of automatically learned masks representing potentially overexposed highlight regions. It helps to reduce the impact of strong specular highlights on diffuse components and at the same time, hallucinates plausible contents in saturated regions. Considering that variation of saturated pixels also contains important cues for inferring surface bumpiness and specular components, we design a two-stream network to extract features from two different branches stacked by HA convolutions and ST convolutions, respectively. These two groups of features are further fused in an attention-based manner to facilitate feature selection of each SVBRDF map. The whole network is trained end to end with a new perceptual adversarial loss which is particularly useful for enhancing the texture details. Such a design also allows the recovered material maps to be disentangled. We demonstrate through quantitative analysis and qualitative visualization that the proposed method is effective to recover clear SVBRDFs from a single casually captured image, and performs favorably against state-of-the-arts. Since we impose very few constraints on the capture process, even a non-expert user can create high-quality SVBRDFs that cater to many graphical applications.

MaterIA: Single Image High‐Resolution Material Capture in the Wild

“…It uses a specific map with radial coordinates to compensate for the degradation of the input by the required flashlight. Guo et al [GLT ∗ 21] mitigate such saturated pixels at the level of network features, by handling the specular highlights within the convolution operation. Tini [Tin20] presents competitive results on the estimation of the normal map, including a Gaussian blending scheme to stitch overlapping tile predictions.…”

Efficient Reflectance Capture with a Deep Gated Mixture-of-Experts

“…Because of its practical value, SVBRDF estimation from a very small number of photographs, often with uncontrolled illumination, has received considerable attention in academia [Aittala et al 2016[Aittala et al , 2015Deschaintre et al 2018;Gao et al 2019;Guo et al 2021;Li et al 2017]. This challenging problem is highly ill-posed, due to the huge gap in the amount of information between the limited input and the 6D output.…”

Deep scene-scale material estimation from multi-view indoor captures

“…Recovering material appearance from a few observations is an ill-posed problem for which machine learning offers practical solutions. By leveraging large datasets of images paired with ground truth material maps, deep convolutional networks can be trained to predict per-pixel material parameters given a single picture of a flat surface patch [21,22,3,23,4,24,25]. Such methods were later extended to predict material, depth and normal maps of isolated objects [5,26] and even of indoor scenes [6] from a single image.…”

“…Our problem is estimating scene-scale material properties from a multi-view dataset under unknown lighting, as opposed to the several successful deep learning methods for estimating SVBRDF maps: These start from one or a few images [21,22,3,4,24,28,30], typically for small planar patches of materials lit by a flash. We tackle the scene-scale material estimation problem by first processing each view with a CNN similar to the one used by Deschaintre et al [3,28]; We explain how we adapted this architecture to our use-case in Sec.…”

“…Learning-based methods for material estimation have focused on pictures of flat surface patches [3,4] or on single images of isolated objects [5] and scenes [6], for which the prediction can be efficiently performed in image-space using convolutional neural networks (CNNs). In contrast, approaches based on inverse rendering compute accurate material parameters in object or texture space [7,8] to benefit from observations from multiple viewpoints.…”