Spatio-Temporal Denoising for Depth Map Sequences

Abstract

This paper proposes a novel strategy for depth video denoising in RGBD camera systems. Depth map sequences obtained by state-of-the-art Time-of-Flight sensors suffer from high temporal noise. Hence, all high-level RGB video renderings based on the accompanied depth maps' 3D geometry like augmented reality applications will have severe temporal flickering artifacts. The authors approached this limitation by decoupling depth map upscaling from the temporal denoising step. Thereby, denoising is processed on raw pixels including uncorrelated pixel-wise noise distributions. The authors' denoising methodology utilizes joint sparse 3D transform-domain collaborative filtering. Therein, they extract RGB texture information to yield a more stable and accurate highly sparse 3D depth block representation for the consecutive shrinkage operation. They show the effectiveness of our method on real RGBD camera data and on a publicly available synthetic data set. The evaluation reveals that the authors' method is superior to state-of-the-art methods. Their method delivers flicker-free depth video streams for future applications.

Depth map artefacts reduction: a review

“…However, the method in [85] is still unable to fill large black pixels. Apart from the local temporal filters and based on the self‐similarity of the depth videos in spatial and temporal domains, video block matching 3D filtering (BM3D/VBM3D) technique is also modified and used by Hach and Seybold [78] to denoise the depth videos from the spatial noise and temporal fluctuations. In this method, block matching jointed between the RGB colour and depth videos is proposed, where the RGB texture assists in searching about similar depth blocks in spatial and temporal domains for enhancing the depth video by removing the spatial and TA.…”