High Quality Photometric Reconstruction Using a Depth Camera

Abstract

In this paper we present a depth-guided photometric 3D reconstruction method that works solely with a depth camera like the Kinect. Existing methods that fuse depth with normal estimates use an external RGB camera to obtain photometric information and treat the depth camera as a black box that provides a low quality depth estimate. Our contribution to such methods are two fold. Firstly, instead of using an extra RGB camera, we use the infra-red (IR) camera of the depth camera system itself to directly obtain high resolution photometric information. We believe that ours is the first method to use an IR depth camera system in this manner. Secondly, photometric methods applied to complex objects result in numerous holes in the reconstructed surface due to shadows and self-occlusions. To mitigate this problem, we develop a simple and effective multiview reconstruction approach that fuses depth and normal information from multiple viewpoints to build a complete, consistent and accurate 3D surface representation. We demonstrate the efficacy of our method to generate high quality 3D surface reconstructions for some complex 3D figurines.

Variational Methods for Normal Integration

“…Another typical situation is when, given both a coarse depth estimate and an accurate normal estimate, one would like to "merge" them in order to create a highquality depth map. Such a problem arises, for instance, when refining the depth map of an RGB-D sensor (e.g., a Kinect) by means of shape-from-shading [42], photometric stereo [25] or shape-from-polarization [32]. In such cases, we may set z 0 to the coarse depth map, and tune λ so as to merge the coarse and fine estimates in the best way.…”

“…In such cases, we may set z 0 to the coarse depth map, and tune λ so as to merge the coarse and fine estimates in the best way. Non-uniform weights may be used, in order to lower the influence of outliers in the coarse depth map [25].…”

Polarized 3D: High-Quality Depth Sensing with Polarization Cues

“…There are numerous existing works that partially or completely reflect these three aspects. Combinations that have been explored previously include: combining a laser scan with PS [31], multi-view stereo with SfS [42] or PS [47,19,7], consumer depth sensing with SfS [44,14,43], and consumer depth sensing with PS [48,15,40]. If high-quality surface normals are not available, fusing a sequence of overlapping depth maps is a popular approach to produce a smooth surface for various interactive applications [17] or large-scale, real-time surface reconstruction [32].…”

“…There are dozens of papers that combine low-quality depth maps with surface normal maps obtained from SfS or PS. Well-regarded papers include [44,14,43] using SfS, and [31,15] using PS. As a complementary technique, we propose the first use of surface normals from polarization to enhance depth maps.…”

Fine-Scale Surface Normal Estimation Using a Single NIR Image

“…They analyze the discriminative characteristics of NIR images and experimentally show the albedo (surface reflectance) simplicity in the NIR wavelength of various materials. In [4,5], they propose the shape refinement methods using the photometric cues in NIR images. They show the high-quality shape recovery results, however they need an additional depth camera to obtain the results.…”