WDRN: A Wavelet Decomposed RelightNet for Image Relighting

Puthussery, Densen; Sethumadhavan, Hrishikesh Panikkasseril; Kuriakose, Melvin; Victor, Jiji Charangatt

doi:10.1007/978-3-030-67070-2_31

Cited by 19 publications

(16 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gafton and Maraz [11] expand the pix2pixHD [25] architecture by adding light direction estimation layers, and train 8 network branches that can relight any image to 1 of 8 illumination directions covered by VIDIT. Puthussery et al [21] perform relighting using wavelet decomposition and propose a new loss responsible for accurate shadow recasting. Their gray loss is calculated on blurred input and ground truth images, stripped from texture information to focus on illumination.…”

Section: Literature Reviewmentioning

confidence: 99%

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

Nikolina¹,

Mustafa²,

Phillipson³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present SILT, a Self-supervised Implicit Lighting Transfer method. Unlike previous research on scene relighting, we do not seek to apply arbitrary new lighting configurations to a given scene. Instead, we wish to transfer the lighting style from a database of other scenes, to provide a uniform lighting style regardless of the input. The solution operates as a two-branch network that first aims to map input images of any arbitrary lighting style to a unified domain, with extra guidance achieved through implicit image decomposition. We then remap this unified input domain using a discriminator that is presented with the generated outputs and the style reference, i.e. images of the desired illumination conditions. Our method is shown to outperform supervised relighting solutions across two different datasets without requiring lighting supervision. The code and pre-trained models can be found here. IntroductionWe propose the problem of lighting transfer where an input image under arbitrary illumination conditions is adapted to match the lighting of a style reference database. Previous approaches to a similar problem -arbitrary image relighting -either make simplifying assumptions about the scene (e.g. a single dominant object or a single light source) or require costly supervision where identical scenes must be captured under a large number of known lighting conditions. In contrast, our lighting transfer approach, SILT, is entirely self-supervised. SILT requires only a training dataset with multiple examples of the same scene and a style reference database. It is not necessary to know the ground truth lighting conditions for the training data, nor is it necessary for the target illumination to be present within the training dataset. This distinction between image relighting and lighting transfer is demonstrated in Fig. 1.The proposed SILT method consists of a two-branch network. During training, the generator sees a number of input images of the same scene under different unknown illuminations. The model attempts to enforce similarity between the illumination conditions of the outputs

show abstract

Section: Literature Reviewmentioning

confidence: 99%

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

Nikolina¹,

Mustafa²,

Phillipson³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They separate these three components in the frequency space. On the other hand, many deep learning-based methods for image relighting [4,5,7] are proposed. Image relighting can be seen as an image-to-image translation problem.…”

Section: Image Relightingmentioning

confidence: 99%

“…Some methods that deal with the ambient conditions especially for the image relighting are developed. Gray loss described in [5] can drive the network to learn the illumination gradient in target domain images. Xu et.al propose a CNN-based method [4] to relight a scene under a new illumination based on five im-Figure 2: The architecture of the proposed multi-modal bifurcated network.…”

Section: Image Relightingmentioning

confidence: 99%

“…Previously, many methods [1,2,3] based on developing visual priors or capture properties of relighted images have achieved impressive performance. Recently, some deep learning-based methods [4,5,6,7] are proposed without explicit inverse rendering steps for estimating scene properties. However, these methods do not consider complex scenes and various ambient conditions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-modal Bifurcated Network for Depth Guided Image Relightin

Yang¹

2021

Preprint

View full text Add to dashboard Cite

Image relighting aims to recalibrate the illumination set?ting in an image. In this paper, we propose a deep learning?based method called multi-modal bifurcated network (MB?Net) for depth guided image relighting. That is, given an image and the corresponding depth maps, a new image with the given illuminant angle and color temperature is gener?ated by our network. This model extracts the image andthe depth features by the bifurcated network in the encoder. To use the two features effectively, we adopt the dynamic dilated pyramid modules in the decoder. Moreover, to in?crease the variety of training data, we propose a novel data process pipeline to increase the number of the training data. Experiments conducted on the VIDIT dataset show that the proposed solution obtains the 1st place in terms of SSIM and PMS in the NTIRE 2021 Depth Guide One-to-one Re?lighting Challenge.

show abstract

“…For scenes, there are workshop tracks (eg [21], [22]), challenges [23], [24] and datasets [25], [26]. Existing work learns image mappings (pure image mappings as in [27]; depth guided, as in [28]; using wavelets, as in [29]; shadow priors, as in [30]). In all cases, methods are learned with paired data (ie images of the same scene under different illuminations), available in the VIDIT dataset [25] and the MIE dataset [26].…”

Section: Related Workmentioning

confidence: 99%

SIRfyN: Single Image Relighting from your Neighbors

Forsyth¹,

Bhattad²,

Asthana³

et al. 2021

Preprint

View full text Add to dashboard Cite

We show how to relight a scene, depicted in a single image, such that (a) the overall shading has changed and (b) the resulting image looks like a natural image of that scene. Applications for such a procedure include generating training data and building authoring environments. Naive methods for doing this fail. One reason is that shading and albedo are quite strongly related; for example, sharp boundaries in shading tend to appear at depth discontinuities, which usually apparent in albedo. The same scene can be lit in different ways, and established theory shows the different lightings form a cone (the illumination cone). Novel theory shows that one can use similar scenes to estimate the different lightings that apply to a given scene, with bounded expected error. Our method exploits this theory to estimate a representation of the available lighting fields in the form of imputed generators of the illumination cone. Our procedure does not require expensive "inverse graphics" datasets, and sees no ground truth data of any kind.Qualitative evaluation suggests the method can erase and restore soft indoor shadows, and can "steer" light around a scene. We offer a summary quantitative evaluation of the method with a novel application of the FID. An extension of the FID allows per-generated-image evaluation. Furthermore, we offer qualitative evaluation with a user study, and show that our method produces images that can successfully be used for data augmentation.

show abstract

WDRN: A Wavelet Decomposed RelightNet for Image Relighting

Cited by 19 publications

References 12 publications

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

SILT: Self-supervised Lighting Transfer Using Implicit Image Decomposition

Multi-modal Bifurcated Network for Depth Guided Image Relightin

SIRfyN: Single Image Relighting from your Neighbors

Contact Info

Product

Resources

About