Sub-pixel data fusion and edge-enhanced distance refinement for 2D/3D images

Abstract

An important field of reasearch in computer vision is the 3D analysis and reconstruction of objects and scenes. A rather new technologie in this context is the Photonic Mixer Device (PMD), based on the time-of-flight principle, which measures full-range distance information in real-time. Unfortunately, PMD-based devices have still limited resolution and provide only IR intensity information.This paper describes a fast algorithmic approach to combine high resolution RGB images with PMD distance data, acquired using a binocular camera setup. The resulting combined RGBZ-data not only enhances the visual result, but also represents a basis for advanced data processing in e. g. object recognition with sub-pixel accuracy. A simple but efficient method is used to detect geometric occlusion caused by the binocular setup, which otherwise will lead to false color assignments.Additionally, we introduce an enhanced filtering technique used for the edge-enhanced distance refinement of the geometry provided by the PMD camera. The technique incorporates a proper handling of boundaries and an iterative refinement approach, which can be used to enhance the 2D/3D-fusion accuracy.

ToF cameras for active vision in robotics

“…A classical solution in the area of object modeling is the use of calibrated stereo rigs. Therefore, initial works were devoted to their comparison with [37] Dynamic object detection and classification Color and light independence PMD Hussmann and Liepert [38] Object pose Easy object/background segmentation PMD Guomundsson et al [39] Known object pose estimation Light independent / Absolute scale SR3 Beder et al [40] Surface reconstruction using patchlets ToF easily combines with stereo PMD Fuchs and May [7] Precise surface reconstruction 3D at high rate SR3/O3D100 (Depth) Dellen et al [5] 3D object reconstruction 3D at high rate SR3 (Depth) Foix et al [6] Kuehnle et al [8] Object recognition for grasping 3D allow geometric primitives search SR3 Grundmann et al [41] Collision free object manipulation 3D at high rate SR3 + stereo Reiser and Kubacki [42] Position based visual servoing 3D is simply obtained / No model needed SR3 (Depth) Gachter et al [43] Object part detection for classification 3D at high rate SR3 Shin et al [44] SR2 Klank et al [45] Mobile manipulation Easy table/object segmentation SR4 Marton et al [46] Object categorization ToF easily combines with stereo SR4 + color Nakamura et al [47] Mobile manipulation Easy table segmentation SR4 + color Saxena et al [9] Grasping unknown objects 3D at high rate SR3 + stereo Zhu et al [48] Short range depth maps ToF easily combines with stereo SR3 + stereo Lindner et al [49] Object segmentation for recognition Easy color registration PMD + color camera Fischer et al [50] Occlusion handling in virtual objects 3D at high rate PMD + color camera…”

“…Another fusion framework is proposed by Lindner et al [49] using calibration and scaling algorithms. They obtain a dense colored depth map using the geometrical points correspondence between the ToF and color cameras by assigning a color to the ToF depth points, and interpolating the depth of the rest of the color camera pixels.…”

Combining Stereo and Time-of-Flight Images with Application to Automatic Plant Phenotyping

“…al. [10] and they provided a biquadratic scheme for this purpose. Due to different viewing positions of ToF and RGB cameras, there are n ToF measurements for z (n ≥ 0) at location (x , y ).…”

“…[1,6,7,10]) lack quantitative results on preserving depth-discontinuity, and most results were qualitative (except [18] which used another 3D scanner to produce pixel-by-pixel depth data). This is partly due to the fact that it is impossible to collect pixel-by-pixel depth data for ground truth.…”

ToF-sensors: New dimensions for realism and interactivity

“…The distance signal can be used to improve the intensity signal and the intensity signal can be used to correct the distance measurement [27]. In [23], depth image refinement techniques for overcoming the low resolution of a PMD sensor, including the enhancement of object boundaries, are discussed.…”