Appearance capture and modeling

In this paper we present a novel method for recovering high-resolution spatially-varying isotropic surface reflectance of a planar exemplar from a flash-lit close-up video sequence captured with a regular hand-held mobile phone. We do not require careful calibration of the camera and lighting parameters, but instead compute a per-pixel flow map using a deep neural network to align the input video frames. For each video frame, we also extract the reflectance parameters, and warp the neural reflectance features directly using the per-pixel flow, and subsequently pool the warped features. Our method facilitates convenient handheld acquisition of spatially-varying surface reflectance with commodity hardware by non-expert users. Furthermore, our method enables aggregation of reflectance features from surface points visible in only a subset of the captured video frames, enabling the creation of high-resolution reflectance maps that exceed the native camera resolution. We demonstrate and validate our method on a variety of synthetic and real-world spatially-varying materials.

Section: Related Workmentioning

confidence: 99%

Deep Reflectance Scanning: Recovering Spatially‐varying Material Appearance from a Flash‐lit Video Sequence

Dong

Peers

et al. 2021

A Versatile Parameterization for Measured Material Manifolds

Soler

Subr

Nowrouzezahrai

2018

A popular approach for computing photorealistic images of virtual objects requires applying reflectance profiles measured from real surfaces, introducing several challenges: the memory needed to faithfully capture realistic material reflectance is large, the choice of materials is limited to the set of measurements, and image synthesis using the measured data is costly. Typically, this data is either compressed by projecting it onto a subset of its linear principal components or by applying non‐linear methods. The former requires many components to faithfully represent the input reflectance, whereas the latter necessitates costly extrapolation algorithms. We learn an underlying, low‐dimensional non‐linear reflectance manifold amenable to rapid exploration and rendering of real‐world materials. We can express interpolated materials as linear combinations of the measured data, despite them lying on an inherently non‐linear manifold. This allows us to efficiently interpolate and extrapolate measured BRDFs, and to render directly from the manifold representation. We exploit properties of Gaussian process latent variable models and use our representation for high‐performance and offline rendering with interpolated real‐world materials.

Photorealistic Material Editing Through Direct Image Manipulation

Zsolnai-Fehér

Wonka

Wimmer

2020

Creating photorealistic materials for light transport algorithms requires carefully fine‐tuning a set of material properties to achieve a desired artistic effect. This is typically a lengthy process that involves a trained artist with specialized knowledge. In this work, we present a technique that aims to empower novice and intermediate‐level users to synthesize high‐quality photorealistic materials by only requiring basic image processing knowledge. In the proposed workflow, the user starts with an input image and applies a few intuitive transforms (e.g., colorization, image inpainting) within a 2D image editor of their choice, and in the next step, our technique produces a photorealistic result that approximates this target image. Our method combines the advantages of a neural network‐augmented optimizer and an encoder neural network to produce high‐quality output results within 30 seconds. We also demonstrate that it is resilient against poorly‐edited target images and propose a simple extension to predict image sequences with a strict time budget of 1–2 seconds per image.