Illumination decomposition for material recoloring with consistent interreflections

Carroll, Robert E.; Ramamoorthi, Ravi; Agrawala, Maneesh

doi:10.1145/1964921.1964938

Cited by 14 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, most typical video sequences do not exhibit illumination changes and one cannot assume a depth channel is available in general. Bousseau et al [2009] allow users to guide the decomposition using scribbles (similar to work on interactive material modeling [Dong et al 2011]), which Carroll et al [2011] use to interactively separate diffuse colored interreflections. However, because their underlying solver uses medium-size image stencils, it takes more than 10 seconds to decompose a half-megapixel image which is too slow for interactive video editing.…”

Section: Related Workmentioning

confidence: 99%

Interactive intrinsic video editing

et al. 2014

View full text Add to dashboard Cite

. Now, editing the textures in the reflectance image does not affect the illumination (c): changes to the brick walls, the roof tiles, and the pathway leading up to the building all maintain the complex illumination of the light through the trees. We encourage readers to zoom into this figure to see the details in our results, and refer them to the accompanying video to see the temporally consistent nature of our decomposition. AbstractSeparating a photograph into its reflectance and illumination intrinsic images is a fundamentally ambiguous problem, and state-of-the-art algorithms combine sophisticated reflectance and illumination priors with user annotations to create plausible results. However, these algorithms cannot be easily extended to videos for two reasons: first, naïvely applying algorithms designed for single images to videos produce results that are temporally incoherent; second, effectively specifying user annotations for a video requires interactive feedback, and current approaches are orders of magnitudes too slow to support this. We introduce a fast and temporally consistent algorithm to decompose video sequences into their reflectance and illumination components. Our algorithm uses a hybrid 2-p formulation that separates image gradients into smooth illumination and sparse reflectance gradients using look-up tables. We use a multi-scale parallelized solver to reconstruct the reflectance and illumination from these gradients while enforcing spatial and temporal reflectance constraints and user annotations. We demonstrate that our algorithm automatically produces reasonable results, that can be interactively refined by users, at rates that are two orders of magnitude faster than existing tools, to produce high-quality decompositions for challenging real-world video sequences. We also show how these decompositions can be used for a number of video editing applications including recoloring, retexturing, illumination editing, and lighting-aware compositing.

show abstract

Section: Related Workmentioning

confidence: 99%

Interactive intrinsic video editing

et al. 2014

View full text Add to dashboard Cite

show abstract

“…In comparison, we make sure that our scribbles only deal with relative characteristics of the scene reflectance, which is significantly easier than determining the absolute color of the local illumination. Carroll et al [2011] use a scribble interface to edit the color of interreflections. While related, our approach deals with effects that are more global and affect large portions of the image, whereas interreflections have a limited spatial extent.…”

Section: Related Workmentioning

confidence: 99%

User-guided white balance for mixed lighting conditions

et al. 2012

View full text Add to dashboard Cite

(a) input with mixed lighting (daylight + neon under the cabinets + low-energy bulbs on ceiling) exhibits unsightly color casts everywhere (b) naive single-light white balance makes the ceiling white, but other color casts remain (c) user indicates regions that are neutral (white strokes) or correct after the single-light white balance (grey strokes) (d) the image is improved, but color variations can still be observed, e.g., on the wooden cabinet (e) user adds marks to specify uniform color, e.g., the cabinet and the wall (f) our final output with no color casts Figure 1: In this photo, the ambient lighting, the cabinet light, and the ceiling lights all have different colors, which produces unpleasant color casts (a). In such situations, the single-light white balance tool provided in all photo editing software only improves a portion of the image, but the result is not satisfying (b). We address this issue by letting users make annotations on the photo. First, they mark objects of neutral color (i.e., white or gray), and regions that look fine after the standard white balance (c). This improves the result, but undesirable color variations are still visible, e.g., on the cabinetry and on the wall (d). Users can indicate that these elements should have a constant color (e), which yields a result free of color cast (f). AbstractProper white balance is essential in photographs to eliminate color casts due to illumination. The single-light case is hard to solve automatically but relatively easy for humans. Unfortunately, many scenes contain multiple light sources such as an indoor scene with a window, or when a flash is used in a tungsten-lit room. The light color can then vary on a per-pixel basis and the problem becomes challenging at best, even with advanced image editing tools.We propose a solution to the ill-posed mixed light white balance problem, based on user guidance. Users scribble on a few regions that should have the same color, indicate one or more regions of neutral color , and select regions where the current color looks correct. We first expand the provided scribble groups to more regions using pixel similarity and a robust voting scheme. We formulate the spatially varying white balance problem as a sparse data interpolation problem in which the user scribbles and their extensions form constraints. We demonstrate that our approach can produce satisfying results on a variety of scenes with intuitive scribbles and without any knowledge about the lights.

show abstract

“…Nonetheless, there are a number of optimization-based methods that disentangle the shading problem by performing additional decompositions. For instance, the illumination image can be separated into direct and multiple indirect components for plausible material coloring (Carroll et al 2011). However, this method requires additional user strokes.…”

Section: Related Workmentioning

confidence: 99%

ShadingNet: Image Intrinsics by Fine-Grained Shading Decomposition

Baslamisli

Das

et al. 2021

Int J Comput Vis

View full text Add to dashboard Cite

In general, intrinsic image decomposition algorithms interpret shading as one unified component including all photometric effects. As shading transitions are generally smoother than reflectance (albedo) changes, these methods may fail in distinguishing strong photometric effects from reflectance variations. Therefore, in this paper, we propose to decompose the shading component into direct (illumination) and indirect shading (ambient light and shadows) subcomponents. The aim is to distinguish strong photometric effects from reflectance variations. An end-to-end deep convolutional neural network (ShadingNet) is proposed that operates in a fine-to-coarse manner with a specialized fusion and refinement unit exploiting the fine-grained shading model. It is designed to learn specific reflectance cues separated from specific photometric effects to analyze the disentanglement capability. A large-scale dataset of scene-level synthetic images of outdoor natural environments is provided with fine-grained intrinsic image ground-truths. Large scale experiments show that our approach using fine-grained shading decompositions outperforms state-of-the-art algorithms utilizing unified shading on NED, MPI Sintel, GTA V, IIW, MIT Intrinsic Images, 3DRMS and SRD datasets.

show abstract

Illumination decomposition for material recoloring with consistent interreflections

Cited by 14 publications

References 34 publications

Interactive intrinsic video editing

Interactive intrinsic video editing

User-guided white balance for mixed lighting conditions

ShadingNet: Image Intrinsics by Fine-Grained Shading Decomposition

Contact Info

Product

Resources

About