Flexible SVBRDF Capture with a Multi‐Image Deep Network

Abstract

Empowered by deep learning, recent methods for material capture can estimate a spatially‐varying reflectance from a single photograph. Such lightweight capture is in stark contrast with the tens or hundreds of pictures required by traditional optimization‐based approaches. However, a single image is often simply not enough to observe the rich appearance of real‐world materials. We present a deep‐learning method capable of estimating material appearance from a variable number of uncalibrated and unordered pictures captured with a handheld camera and flash. Thanks to an order‐independent fusing layer, this architecture extracts the most useful information from each picture, while benefiting from strong priors learned from data. The method can handle both view and light direction variation without calibration. We show how our method improves its prediction with the number of input pictures, and reaches high quality reconstructions with as little as 1 to 10 images ‐ a sweet spot between existing single‐image and complex multi‐image approaches.

MaterialGAN

“…We demonstrate that our GAN-based optimization framework produces high-quality SVBRDF reconstructions from a small number (3-7) images captured under flash illumination using hand-held mobile phones, and improves upon previous state-of-the-art methods [Deschaintre et al 2019;Gao et al 2019]. In particular, it produces cleaner, more realistic material maps that better reproduce the appearance of the captured material under both input and novel lighting.…”

“…For our real results, we use a hand-held mobile phone to capture images with flash, resulting in a collocated camera and point light illumination. Similar to previous work [Deschaintre et al 2019;Hui et al 2017], we use a paper frame to register the multiple images. We add markers to the frame to improve camera pose estimation.…”

“…More recently, learning-based approaches have demonstrated remarkable results for reconstructing SVBRDFs from one [Deschaintre et al 2018;Li et al 2018] or more images [Deschaintre et al 2019]. While these methods use rendering-based losses (similar to the inverse rendering approaches) during training, at test time they predict SVBRDFs from images using a single feed-forward pass through a deep network.…”

“…These methods train deep convolutional neural networks with large datasets of artistically created SVBRDFs, and with a combination of losses that evaluate the difference in material maps and renderings from the dataset ground truth. Deschaintre [2019] extended the single-shot approach to multiple images. The key idea is to extract features from the input images with a shared encoder, max-pooling the features and decoding the final maps from the pooled features.…”

“…SVBRDF reconstruction on synthetic data. We demonstrate results on synthetic SVBRDFs, one from [Deschaintre et al 2019] (top) and one from the Adobe Stock Material dataset (bottom). We are able to accurately reconstruct these materials from 7 input images (one input shown).…”