Learning efficient illumination multiplexing for joint capture of reflectance and shape

Kang, Kaizhang; Xie, Cihui; He, Chengan; Yi, Mingqi; Gu, Minyi; Chen, Zimin; Zhou, Kun; Wu, Hongzhi

doi:10.1145/3355089.3356492

Cited by 45 publications

(36 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, mixed-domain networks are proposed, to jointly and automatically optimize physical lighting patterns along with the computational reconstruction. Efficient capture of pixel-independent anisotropic reflectance is achieved on planar samples [Kang et al 2018] and non-planar ones [Kang et al 2019].…”

Section: Point Light(s)mentioning

confidence: 99%

“…We validate our reconstructions against photographs, as well as the counterpart captured with a high-end lightstage [Kang et al 2019]. The results are also compared against one state-of-the-art technique on mobile scanning of non-planar appearance [Nam et al 2018].…”

Section: Introductionmentioning

confidence: 98%

“…Priors are required to trade the spatial resolution for the angular accuracy [Nam et al 2018;Riviere et al 2016]. On the other hand, illumination multiplexing devices like lightstages can acquire complex appearance with high performance, by independently controlling a large number of light sources [Kang et al 2019;Tunwattanapong et al 2013]. However, related techniques strongly exploit the fixed view condition(s) while the illumination changes.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Free-form scanning of non-planar appearance with neural trace photography

Kang

Zhu

et al. 2021

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

We propose neural trace photography, a novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance. Our key insight is that free-form appearance scanning can be cast as a geometry learning problem on unstructured point clouds, each of which represents an image measurement and the corresponding acquisition condition. Based on this connection, we carefully design a neural network, to jointly optimize the lighting conditions to be used in acquisition, as well as the spatially independent reconstruction of reflectance from corresponding measurements. Our framework is not tied to a specific setup, and can adapt to various factors in a data-driven manner. We demonstrate the effectiveness of our framework on a number of physical objects with a wide variation in appearance. The objects are captured with a light-weight mobile device, consisting of a single camera and an RGB LED array. We also generalize the framework to other common types of light sources, including a point, a linear and an area light.

show abstract

Section: Point Light(s)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Free-form scanning of non-planar appearance with neural trace photography

Kang

Zhu

et al. 2021

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Related to our process of learning a decoder, Genetic Programming (GP) has previously been employed to learn new analytic BRDF models that better describe specific materials [BLPW14]. Many recent works have proposed employing deep learning to efficiently fit a parametric BRDF model instead of employing traditional non‐linear optimization [AAL16, LDPT17, LSC18, LXR∗18, DAD∗18, MHRK19, BL19, KXH∗19]. Many of these methods are derived from U‐Net or auto‐encoder architectures [KCW∗18, GLD∗19].…”

Section: Related Workmentioning

confidence: 99%

Unified Neural Encoding of BTFs

Rainer

Ghosh

Jakob

et al. 2020

Computer Graphics Forum

View full text Add to dashboard Cite

Realistic rendering using discrete reflectance measurements is challenging, because arbitrary directions on the light and view hemispheres are queried at render time, incurring large memory requirements and the need for interpolation. This explains the desire for compact and continuously parametrized models akin to analytic BRDFs; however, fitting BRDF parameters to complex data such as BTF texels can prove challenging, as models tend to describe restricted function spaces that cannot encompass real‐world behavior. Recent advances in this area have increasingly relied on neural representations that are trained to reproduce acquired reflectance data. The associated training process is extremely costly and must typically be repeated for each material. Inspired by autoencoders, we propose a unified network architecture that is trained on a variety of materials, and which projects reflectance measurements to a shared latent parameter space. Similarly to SVBRDF fitting, real‐world materials are represented by parameter maps, and the decoder network is analog to the analytic BRDF expression (also parametrized on light and view directions for practical rendering application). With this approach, encoding and decoding materials becomes a simple matter of evaluating the network. We train and validate on BTF datasets of the University of Bonn, but there are no prerequisites on either the number of angular reflectance samples, or the sample positions. Additionally, we show that the latent space is well‐behaved and can be sampled from, for applications such as mipmapping and texture synthesis.

show abstract

“…It is thus extremely difficult to simultaneously recover the unknown material and object shape, even under known lighting conditions. To address this challenge, sophisticated hardware such as light stages [6], coaxial lights [13], and near-field light stages [17] have been designed. Though these methods achieve highly accurate results, the setups are expensive and complicated.…”

Section: Introductionmentioning

confidence: 99%

Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials

Zhou

et al. 2020

IEEE Trans. on Image Process.

115

View full text Add to dashboard Cite

We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo (MVPS) technique that works for general isotropic materials. Our algorithm is suitable for perspective cameras and nearby point light sources. Our data capture setup is simple, which consists of only a digital camera, some LED lights, and an optional automatic turntable. From a single viewpoint, we use a set of photometric stereo images to identify surface points with the same distance to the camera. We collect this information from multiple viewpoints and combine it with structure-from-motion to obtain a precise reconstruction of the complete 3D shape. The spatially varying isotropic bidirectional reflectance distribution function (BRDF) is captured by simultaneously inferring a set of basis BRDFs and their mixing weights at each surface point. In experiments, we demonstrate our algorithm with two different setups: a studio setup for highest precision and a desktop setup for best usability. According to our experiments, under the studio setting, the captured shapes are accurate to 0.5 millimeters and the captured reflectance has a relative root-mean-square error (RMSE) of 9%. We also quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset, which is publicly available for inspiring future research.

show abstract

Learning efficient illumination multiplexing for joint capture of reflectance and shape

Cited by 45 publications

References 30 publications

Free-form scanning of non-planar appearance with neural trace photography

Free-form scanning of non-planar appearance with neural trace photography

Unified Neural Encoding of BTFs

Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials

Contact Info

Product

Resources

About