Real‐time depth enhancement by fusion for RGB‐D cameras

Abstract

Abstract. This paper presents a real-time refinement procedure for depth data acquired by RGB-D cameras. Data from RGB-D cameras suffers from undesired artifacts such as edge inaccuracies or holes due to occlusions or low object remission. In this work, we use recent depth enhancement filters intended for Time-of-Flight cameras, and extend them to structured light based depth cameras, such as the Kinect camera. Thus, given a depth map and its corresponding 2-D image, we correct the depth measurements by separately treating its undesired regions. To that end, we propose specific confidence maps to tackle areas in the scene that require a special treatment. Furthermore, in the case of filtering artifacts, we introduce the use of RGB images as guidance images as an alternative to real-time state-of-the-art fusion filters that use grayscale guidance images. Our experimental results show that the proposed fusion filter provides dense depth maps with corrected erroneous or invalid depth measurements and adjusted depth edges. In addition, we propose a mathematical formulation that enables to use the filter in real-time applications.

Unified multi-lateral filter for real-time depth map enhancement

“…Tiny depth variations are mainly located along the low-weighted depth boundaries by the credibility map, where the UML filter behaves as the PWAS filter. We note that the reported SSIM measures are slightly different from those reported in our previous works [13,41]. The reason is that the nonvalid/background depth pixels have not been considered in this evaluation.…”

“…In order to achieve a close to real-time performance, we proposed in [41] to not consider the full 3-D colour information but only the pixelwise RGB channel that best describes the 2-D edge. Although the processing time is slightly increased, this UML filter extension that we named RGB-D filter addresses the edge blurring artefact owing to RGB to grayscale transformation.…”

“…To this end, we advise in [41] on how the reliability weight β in (7) has to be set to deal with depth discontinuities along object boundaries. An alternative UML configuration to enhance depth data from passive stereo sensing systems was also proposed in [45].…”

3DBodyTex: Textured 3D Body Dataset

“…The two pieces of information combined provide more dimensions to model and process, e.g. [9,26,16]. This richer information is desired in several scenarios.…”

Robust depth enhancement and optimization based on advanced multilateral filters

“…However, its computational complexity is high and its estimation accuracy would be fail in texture-less and occluded regions. Recently, the low-cost structured-light RGB-D cameras have been used to capture high-resolution color images and low-resolution depth maps [10]. Thus, depth map upsampling [11,12] followed by its enhancement [13] becomes an inevitable task because the quality of the DIBR process heavily depends on the accuracy of depth information.…”