Learning Morphological Operators for Depth Completion

Abstract

Abstract. Depth images generated by direct projection of LiDAR point clouds on the image plane suffer from a great level of sparsity which is difficult to interpret by classical computer vision algorithms. We propose a method for completing sparse depth images in a semantically accurate manner by training a novel morphological neural network. Our method approximates morphological operations by Contraharmonic Mean Filter layers which are easily trained in a contemporary deep learning framework. An early fusion U-Net architecture then combines dilated depth channels and RGB using multi-scale processing. Using a large scale RGB-D dataset we are able to learn the optimal morphological and convolutional filter shapes that produce an accurate and fully sampled depth image at the output. Independent experimental evaluation confirms that our method outperforms classical image restoration techniques as well as current state-of-the-art neural networks. The resulting depth images preserve object boundaries and can easily be used to augment various tasks in intelligent vehicles perception systems.

Learning Topology From Synthetic Data for Unsupervised Depth Completion

“…We compare our unsupervised method (italized, row 3 in the table) against supervised methods on the KITTI depth completion benchmark. We also note that while most supervised methods still do better, our approach surpasses some supervised methods: [5] across all metrics, [6] on MAE, iMAE, and iRMSE metrics and [21] on iMAE. This demonstrates the potential of our method in closing the gap between supervised and unsupervised methods.…”

“…are the best performing method on the unsupervised setting, we note that the benchmark is still dominated by supervised methods. However, our approach shows promise as we surpass some supervised methods: [5] across all metrics, [6] on MAE, iMAE, and iRMSE metrics and [21] on iMAE. We hope that our approach will lay the foundation for the push to close the gap between supervised and unsupervised learning frameworks for the depth completion task.…”

Dense Depth Posterior (DDP) From Single Image and Sparse Range

“…[7] treats the binary validity map as a confidence map and adapts normalized convo-lution for confidence propagation through layers. [5] implements an approximation of morphological operators using the contra-harmonic mean (CHM) filter [23] and incorporates it as a layer in a U-Net architecture for depth completion. [4] proposes a deep recurrent auto-encoder to mimic the optimization procedure of compressive sensing for depth completion, where the dictionary is embedded in the neural network.…”

“…where d j = φ(z j , I j ; w) and L u is the loss defined in Eq. (5). Note that, the above summation term is the instantiation for P raw (I |I, d), which can also be replaced by the SSIM counterpart.…”

Deep Architecture With Cross Guidance Between Single Image and Sparse LiDAR Data for Depth Completion

“…There are no depth data for most pixels when the LiDAR map is transformed to be aligned with an RGB camera [1]. Previous studies considered sparse inputs as an inpainting problem using classical image processing methods such as hand-crafted kernels or interpolation [22]- [25]. Recently, many approaches that employ deep learning and CNNs have shown successful results.…”