A new multi-lateral filter for real-time depth enhancement

Abstract

We present an adaptive multi-lateral filter for real-time low-resolution depth map enhancement. Despite the great advantages of Time-of-Flight cameras in 3-D sensing, there are two main drawbacks that restricts their use in a wide range of applications; namely, their fairly low spatial resolution, compared to other 3-D sensing systems, and the high noise level within the depth measurements. We therefore propose a new data fusion method based upon a bilateral filter. The proposed filter is an extension the pixel weighted average strategy for depth sensor data fusion. It includes a new factor that allows to adaptively consider 2-D data or 3-D data as guidance information. Consequently, unwanted artefacts such as texture copying get almost entirely eliminated, outperforming alternative depth enhancement filters. In addition, our algorithm can be effectively and efficiently implemented for real-time applications.

Depth Enhancement by Fusion for Passive and Active Sensing

“…Though bilateral filtering is known to be time consuming, its latest fast implementations based on data quantization and downsampling [12,13], also applicable to both PWAS and UML filters as demonstrated in [5], enable a highperformance. Thus, in order to ensure a real-time performance, we propose to downsample by a factor of s the credibility maps Q D and Q I preserving such regions that require special treatment.…”

“…In order to significantly reduce this artifact, the same authors proposed in [5] the Unified Multi-Lateral (UML) filter. The UML filter combines two PWAS filters where the output J 3 of the second one has both spatial and range kernels acting onto the same data source D. In addition, they suggested to use the credibility map Q as a blending function, i.e., β = Q, hence, depth pixels with high reliability are not influenced by the 2-D data avoiding texture copying as follows…”

“…However, we realise that if we follow the recommendations of Garcia et. al [5] and we set the blending function β equal to the credibility map Q, edge blurring will appear when filtering low reliable depth pixels if no 2-D edge is present. Indeed, this situation occurs when foreground and background objects share the same intensity value, which often occurs when considering grayscale images.…”

“…We propose to look into approaches of fusion by filtering tested and proven on Time-of-Flight (ToF) cameras [3][4][5][6]. Our main goal is to generalize such approach and define a global framework that may be adapted to other active sensors, specifically an RGB-D consumer camera, but also to more traditional passive ones, such as a dense stereo camera.…”

Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations

“…This category of methods is based on the assumption that there is a correspondence between the edges of a depth map and the edges of another modality of a better quality, often chosen to be a 2D intensity image of the same scene [19], [20], [21], [22], [23], [24], [26]. Such an image is considered to be a guidance image whose properties, in terms of structure, signal to noise ratio (SNR) and resolution, can be transferred to its corresponding depth map in order to obtain an enhanced version.…”