Repurposing a Relighting Network for Realistic Compositions of Captured Scenes

Abstract

Inria (a) Or i g i n a l I ma g e (b) S e l e c t i o n (c) Co mp o s i t e Re s u l t Figure 1: Example of a composition of two captured historical landmarks using our method. We extract the geometry of one (b) from a multi-view dataset, and import it into the other (a). Our method ensures coherent treatment of lighting and shadows, producing a realistic result (c).

Shadow Layers for Participating Media

“…We reckon that shadow layers can also be useful outside of the compositing pipeline, and apply to machine learning approaches where networks are trained to infer or remove shadow in photographs [WLY18, LS19], or perform image‐based relighting with plausible shadows [PGZ*19, PMGD21, NPD20]. Compared to binary masks, our representation is linear and should better perform in linear operations such as convolution.…”

“…Alternatively, neural networks incorporate strong priors that help the disambiguation [WLY18,LS19]. Philip et al have used them in image-based rendering to plausibly reproduce shadows under new lighting conditions [PGZ * 19, PMGD21], or after an extraneous object is added into the scenery [NPD20]. While these methods could be applied to extract approximate shadows in renders, we propose to reuse information from the lighting simulation for an exact result.…”

Intrinsic Harmonization for Illumination-Aware Image Compositing

“…Given the complexity of image relighting, prior methods mainly focus on a specific use case such as portraits [Pandey et al 2021;Yeh et al 2022] or outdoor structures [Griffiths et al 2022]. These methods rely on large-scale, difficult-to-obtain datasets, or multi-view scenes [Nicolet et al 2020;Philip et al 2019Philip et al , 2021, in order to achieve realistic results.…”