A Generative Framework for Image‐based Editing of Material Appearance using Perceptual Attributes

Abstract

Single‐image appearance editing is a challenging task, traditionally requiring the estimation of additional scene properties such as geometry or illumination. Moreover, the exact interaction of light, shape and material reflectance that elicits a given perceptual impression is still not well understood. We present an image‐based editing method that allows to modify the material appearance of an object by increasing or decreasing high‐level perceptual attributes, using a single image as input. Our framework relies on a two‐step generative network, where the first step drives the change in appearance and the second produces an image with high‐frequency details. For training, we augment an existing material appearance dataset with perceptual judgements of high‐level attributes, collected through crowd‐sourced experiments, and build upon training strategies that circumvent the cumbersome need for original‐edited image pairs. We demonstrate the editing capabilities of our framework on a variety of inputs, both synthetic and real, using two common perceptual attributes (Glossy and Metallic), and validate the perception of appearance in our edited images through a user study.

In‐the‐wild Material Appearance Editing using Perceptual Attributes

“…We compare our results against the method of Delanoy et al [DLC * 22]. We show results reconstructing the input image using the crowd‐sourced perceptual attribute as the input, and editing the input using a different target perceptual attribute.…”

“…Delanoy et al [DLC * 22] output is highly dependent to the estimated normal map of the input image. A low‐frequency estimation of the normal map yields dull edits of material appearance.…”

“…This allows us to keep high‐frequency details in the generated image that are lost otherwise [KW13, HMP * 17]. However, in image editing tasks that manipulate the latent space, adding skip connections hampers the editability of the model, where only the input image can be reconstructed [CCK * 18, HZK * 19, DLC * 22]. To bridge this gap, we send information from the encoder to the decoder by selectively removing unnecessary data using Selective Transfer Units (STU) memory cells [LDX * 19].…”

“…This approach is not only tedious but also hinders the training process. We follow the work from Delanoy et al [DLC * 22], which is based on training a two‐step generative framework using a wide dataset composed of a large variety of images, paired with high‐level perceptual ratings obtained through user studies. However, this method needs additional geometry information (as a normal map) of the target object.…”

Bumpy Appearance Editing of Object Surfaces in Digital Images

“…1062-3701/2022/66(5)/050403/15/$25.00 illumination and surface reflectance properties to determine the perception of gloss by the observer [3][4][5][6][7][8][9][10].…”