A novel framework for highlight reflectance transformation imaging

Abstract

We propose a novel pipeline and related software tools for processing the multi-light image collections (MLICs) acquired in different application contexts to obtain shape and appearance information of captured surfaces, as well as to derive compact relightable representations of them. Our pipeline extends the popular Highlight Reflectance Transformation Imaging (H-RTI) framework, which is widely used in the Cultural Heritage domain. We support, in particular, perspective camera modeling, per-pixel interpolated light direction estimation, as well as light normalization correcting vignetting and uneven non-directional illumination. Furthermore, we propose two novel easy-to-use software tools to simplify all processing steps. The tools, in addition to support easy processing and encoding of pixel data, implement a variety of visualizations, as well as multiple reflectance-model-fitting options. Experimental tests on synthetic and real-world MLICs demonstrate the usefulness of the novel algorithmic framework and the potential benefits of the proposed tools for end-user applications.

State‐of‐the‐art in Multi‐Light Image Collections for Surface Visualization and Analysis

“…In free‐form setups, instead, the light source is freely moved in space with hand‐held or robotics setups [MDA02, CGS06, MVSL05, KK13]. This approach is more flexible but requires more calibration efforts to deal with non‐uniformity of light intensity and the variable spacing of positioning [GCD∗18], and more effort to design interfaces to navigate in the data. For these reasons, physical constraints are often applied to approximate a uniform sampling of the hemisphere in a free‐form setting [CHI19], or, alternatively, a regular light distribution is simulated through interpolation before further processing or interaction [PCS18].…”

“…A large variety of solutions have been proposed, including assuming mechanically calibrated arrangements for domes and rings [RTF∗04], using known calibration objects [GCD∗18, Mac15, CCC08, MDA02, GCD∗17], and/or autocalibration techniques that reconstruct illumination parameters by making assumptions on the image reflectance field [WMTG05, XDW15, PF14, GCD∗18] or on the light form factor [ASSS14, USTY15, PCGG16, XSJ∗15]. Ackermann et al [AFG13] present a validation through error statistics of both a forward and backward geometric point light source calibration by using sets of different numbers of reflective spheres, showing that significant errors are present even in common use cases just for the identification of light position.…”

“…In order to cope with more complex real‐world lights, some methods virtually use a different per‐pixel light direction. Since the field is smooth and slowly varying, this per‐pixel lighting information is not usually stored at image resolution, but approximated by a low‐degree polynomial function [GCD∗18]. It should be noted that in these methods the incident lighting direction is expressed relative to a particular reference object (e.g., a constant‐depth plane), and not to the imaged object, whose actual depth is generally a result of further processing.…”

“…Finally, note that different terms are often used to refer to the same light parameterization. For instance, the measured color as a function of incoming light direction is sometimes called Light transport function [HWBC15] or Appearance profile [GCD∗18] and its discretization on a grid has been called Observation map [Ike18].…”

A Complete Framework Operating Spatially-Oriented Rti in a 3d/2d Cultural Heritage Documentation and Analysis Tool

“…On one hand, Multi-view and imagebased modeling (CRP and SfM) are periodically discussed regarding their gain of completeness and/or accuracy in necessary critical overviews (Remondino et al, 2014), whereas they can be exploited nowadays as a stable and reliable technique. On the other hand, Multi-light and RTI's techniques are massively developing from the original open-source code (Malzbender et al, 2001, Mudge et al, 2006, with numerous enhancements concerning the fitting-viewing side (Palma et al, 2010, Giachetti et al, 2017a and going to multispectral (Hanneken, 2014, Giachetti et al, 2017b and new devices (Schuster et al, 2014) support.…”