Deep appearance modeling: A survey

Dong, Yue

doi:10.1016/j.visinf.2019.07.003

Cited by 40 publications

(19 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Acquiring material data from physical measurements is the goal of a broad range of methods. Please refer to surveys [Dong 2019;Guarnera et al 2016;Weyrich et al 2009] for more comprehensive introduction to the related works.…”

Section: Related Workmentioning

confidence: 99%

MaterialGAN

et al. 2020

View full text Add to dashboard Cite

We address the problem of reconstructing spatially-varying BRDFs from a small set of image measurements. This is a fundamentally under-constrained problem, and previous work has relied on using various regularization priors or on capturing many images to produce plausible results. In this work, we present MaterialGAN , a deep generative convolutional network based on StyleGAN2, trained to synthesize realistic SVBRDF parameter maps. We show that MaterialGAN can be used as a powerful material prior in an inverse rendering framework: we optimize in its latent representation to generate material maps that match the appearance of the captured images when rendered. We demonstrate this framework on the task of reconstructing SVBRDFs from images captured under flash illumination using a hand-held mobile phone. Our method succeeds in producing plausible material maps that accurately reproduce the target images, and outperforms previous state-of-the-art material capture methods in evaluations on both synthetic and real data. Furthermore, our GAN-based latent space allows for high-level semantic material editing operations such as generating material variations and material morphing.

show abstract

Section: Related Workmentioning

confidence: 99%

MaterialGAN

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The approaches that handle uncontrolled lighting Wu et al 2016a; are beyond the scope of this paper. Interested readers are directed to excellent surveys on acquisition techniques [Dong 2019;Guarnera et al 2016;Weinmann and Klein 2015;Weyrich et al 2009]. In general, densely sampling the 4D view-illumination domain with a camera produces high-quality results in manually designed [Dana et al 1999;Lawrence et al 2006] or even learned appearance representations [Gao et al 2020].…”

Section: Related Workmentioning

confidence: 99%

Free-form scanning of non-planar appearance with neural trace photography

Kang

Zhu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

We propose neural trace photography, a novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance. Our key insight is that free-form appearance scanning can be cast as a geometry learning problem on unstructured point clouds, each of which represents an image measurement and the corresponding acquisition condition. Based on this connection, we carefully design a neural network, to jointly optimize the lighting conditions to be used in acquisition, as well as the spatially independent reconstruction of reflectance from corresponding measurements. Our framework is not tied to a specific setup, and can adapt to various factors in a data-driven manner. We demonstrate the effectiveness of our framework on a number of physical objects with a wide variation in appearance. The objects are captured with a light-weight mobile device, consisting of a single camera and an RGB LED array. We also generalize the framework to other common types of light sources, including a point, a linear and an area light.

show abstract

“…Early work [48], [49] leveraged Photometric Stereo [50] and SVBRDF manifold bootstrapping [51] for surface geometry reconstruction, while newer methods exploit the power of deep neural networks. Recent surveys by Guarnera [52] and Dong [53] contain the most relevant approaches. While our method is not meant to estimate the SVBRDF properties of a material, it can be used in combination with those techniques to create larger material assets.…”

Section: Svbrdf Estimationmentioning

confidence: 99%

Neural Photometry-guided Visual Attribute Transfer

Rodriguez-Pardo,

Garces

2021

Preprint

View full text Add to dashboard Cite

We present a deep learning-based method for propagating spatially-varying visual material attributes (e.g. texture maps or image stylizations) to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy, making the transfer robust to novel illumination conditions and affine deformations. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain; we showcase a semantic segmentation, a normal map, and a stylization. Following an image analogies approach, the method only requires the training data to contain the same visual structures as the input guidance. Our approach works at interactive rates, making it suitable for material edit applications. We thoroughly evaluate our learning methodology in a controlled setup providing quantitative measures of performance. Last, we demonstrate that training the model on a single material is enough to generalize to materials of the same type without the need for massive datasets.

show abstract

Deep appearance modeling: A survey

Cited by 40 publications

References 52 publications

MaterialGAN

MaterialGAN

Free-form scanning of non-planar appearance with neural trace photography

Neural Photometry-guided Visual Attribute Transfer

Contact Info

Product

Resources

About