Light-Field Appearance Editing based on Intrinsic Decomposition

Abstract

The authors present a framework for image-based surface appearance editing for light-field data. Their framework improves over the state of the art without the need for a full “inverse rendering,” so that full geometrical data, or presence of highly specular or reflective surfaces are not required. It is robust to noisy or missing data, and handles many types of camera array setup ranging from a dense light field to a wide-baseline stereo-image pair. They start by extracting intrinsic layers from the light-field image set maintaining consistency between views. It is followed by decomposing each layer separately into frequency bands, and applying a wide range of “band-sifting” operations. The above approach enables a rich variety of perceptually plausible surface finishing and materials, achieving novel effects like translucency. Their GPU-based implementation allow interactive editing of an arbitrary light-field view, which can then be consistently propagated to the rest of the views. The authors provide extensive evaluation of our framework on various datasets and against state-of-the-art solutions.

Interactive Photo Editing on Smartphones via Intrinsic Decomposition

“…Our specularity removal step is motivated from the above observation. Further, in order to make our method robust against color artifacts we use image intensity L instead of luminance for the above [BSM ∗ 18]. The chromaticity C of the input image I (with color channels R, G , and B ) is processed separately to handle missing color information for saturated specular pixels.…”

“…Our material editing framework is based on the work of Beigpour et al [BSM ∗ 18], where the authors modify the intensity of albedo, shading, and specularity using band‐sifting filters [BBPA15]. The modified intrinsic layers are merged to form the output image ( I out ) with edited appearance,…”

“…The Retinex theory by Land and McCann proved to be a crucial finding, which became part of many following algorithms as a prior [LM71]. In the course of previous decades, intrinsic decomposition algorithms have been proposed for image [TFA05, BBS14, BM15, ZTD ∗ 12, ZKE15, KPSL16, CZL18, MCZ ∗ 18, LS18, LXR ∗ 18, LSR ∗ 20], video [YGL ∗ 14, BST ∗ 14, MZRT16], multiple‐views [LBD13, DRC ∗ 15, MQD ∗ 17] and light‐fields [GEZ ∗ 17, AG16, AJSG18, BSM ∗ 18]. A survey covering many of these algorithms is provided by Bonneel et al [BKPB17].…”

“…The algorithm by Alperovich et al [AG16] is designed for light‐fields but cannot be used for a single image. The method of Beigpour et al [BSM ∗ 18] is applicable for a single image and, like ours, removes specularities in a pre‐processing step. However, for specularity extraction, they do not consider chroma channels leading to artifacts in highly saturated image regions.…”

A Learned Compact and Editable Light Field Representation

“…Zhang et al [10] decompose the central view into a set of layers at different depths, so existing patch-based edits can be applied to them. Recently, Beigpour et al [34] present an intrinsic decomposition framework for editing the appearance of surfaces through various band shift operators. Aiming to exploit the 4D information in a light field, Jarabo et al [8] propose and study novel interfaces and workflows.…”