Deep image-based relighting from optimal sparse samples

Abstract

a) Input images under directional lights (c) Our result under a novel directional light (b) Ground truth under a novel directional light (d) Our results under environment map illumination Fig. 1. We propose a learning-based method that takes only five images of a scene under directional lights (a, light directions marked on circle in red) and reconstructs its appearance (c) under a novel directional light in the upper hemisphere (marked in orange). Our method trains a fully-convolutional neural network to jointly learn the optimal input light directions and relighting function for any scene. The network can reconstruct even high-frequency patterns like specular shading and cast shadows (insets in c) and produces photorealistic relighting results that closely match the ground truth (b). Moreover, by generating images for every direction in the upper hemisphere, our method can be used to relight scenes under environment map illumination (d).We present an image-based relighting method that can synthesize scene appearance under novel, distant illumination from the visible hemisphere, from only five images captured under pre-defined directional lights. Our method uses a deep convolutional neural network to regress the relit image from these five images; this relighting network is trained on a large synthetic dataset comprised of procedurally generated shapes with real-world reflectances. We show that by combining a custom-designed sampling network with the relighting network, we can jointly learn both the optimal input light directions and the relighting function. We present an extensive evaluation of our network, including an empirical analysis of reconstruction quality, optimal lighting configurations for different scenarios, and alternative network architectures. We demonstrate, on both synthetic and real scenes, that our method is able to reproduce complex, high-frequency lighting effects like specularities and cast shadows, and outperforms other image-based relighting methods that require an order of magnitude more images.

State‐of‐the‐art in Multi‐Light Image Collections for Surface Visualization and Analysis

“…Looking at several works [ZD14, PLGM∗17a], it seems that, using a similar number of coefficients, PTM/HSH and DMD provide similar results. The results reported by Xu et al [XSHR18] show that the CNN based relighting presented may provide higher PSNR than PTM and BRDF based methods [HS17].…”

“…This results in the typical “blended” shadows, less evident in the local interpolation of dense sampling, but still clearly visible. Global CNN‐based relighting methods [XSHR18] solve this problem, but create sharp shadows with hallucinated shape, not corresponding to the real one (Fig. 13).…”

“…The blended shadows artifacts obtained in interpolation‐based relighting (left). The method proposed by Xu et al [XSHR18] avoids it by using a global CNN based relighting, but the shadow shape is clearly different from the real one. Courtesy of Xu et al .…”

“…We penalize the global CNN based approach due to detail hallucination. According to the evaluation of Xu et al [XSHR18], we considered not very good, at the current state of work, both photorealism and accuracy for the BRDF based methods, mainly because of the difficulty of their practical application to common settings (multi‐material object with complex shapes and limited numbers of available views). We expect that, at least for selected object and material classes, BRDF based method providing photorealistic and accurate results will emerge in the near future.…”

“…In contrast to the methods that recover the relighted image independently per pixel, Xu et al [XSHR18] have proposed a method to learn the behaviour for entire image patches using specialized Convolutional Neural Networks (CNNs). Two networks are trained, one to select optimal light sampling (that, therefore, is then fixed with a certain tolerance), and the other to relight the scene given a novel light direction.…”