Scene reassembly after multimodal digitization and pipeline evaluation using photorealistic rendering

Abstract

Transparent objects require acquisition modalities that are very different from the ones used for objects with more diffuse reflectance properties. Digitizing a scene where objects must be acquired with different modalities requires scene reassembly after reconstruction of the object surfaces. This reassembly of a scene that was picked apart for scanning seems unexplored. We contribute with a multimodal digitization pipeline for scenes that require this step of reassembly. Our pipeline includes measurement of bidirectional reflectance distribution functions and high dynamic range imaging of the lighting environment. This enables pixelwise comparison of photographs of the real scene with renderings of the digital version of the scene. Such quantitative evaluation is useful for verifying acquired material appearance and reconstructed surface geometry, which is an important aspect of digital content creation. It is also useful for identifying and improving issues in the different steps of the pipeline. In this work, we use it to improve reconstruction, apply analysis by synthesis to estimate optical properties, and to develop our method for scene reassembly.

Using a Robotic Arm for Measuring BRDFs

“…It could also stem from a bias in the MERL dataset. Reconstructions of other materials with less defined highlights were much better: BRDFs of white and gray cardboard and white cloth were measured with our system in our previous work [18]. This work did not present the calibration and unbiased interpolation techniques covered here, but it does provide a larger selection of measured BRDF data delivered by the system and validated to some extent through comparison of rendered images with photographs.…”

“…These systems employ a ring-shaped rotation stage for holding either a light source [1,15] or a spectroradiometer [16]. Instead of holding the sample, we let the robot hold a camera, as we can then use the robot in a wider range of applications [18]. The facility at Fraunhofer [8] is similar to ours but more elaborate.…”

“…It is thus a plausible estimate of the missing values, but confined to the data-variation in the MERL database. More info on this biased interpolation method is available in our previous work [18].…”

Single-Shot Analysis of Refractive Shape Using Convolutional Neural Networks

“…Photorealistic rendering of transparent materials is challenging since visually significant paths can be very long when we account for both refraction and reflection. Based on observations in the seemingly first study with quantitatively verified photorealistic rendering of glass objects [30], we use full path tracing with deep paths. In path tracing, refraction or reflection is chosen probabilistically based on Fresnel reflectance, which depends on the angle of incidence of the light [23].…”

Material-Based Segmentation of Objects

“…Our study is inspired by researchers reconstructing the shape of glass objects using a dataset of synthetic images [28]. This work was based on an earlier investigation verifying that physically based rendering can produce images that are pixelwise comparable to photographs of specifically glass [27]. Considering this information, we wish to explore the potential of rendering of different materials with complex radiometric properties.…”