Revisiting Sparsity Invariant Convolution: A Network for Image Guided Depth Completion

Abstract

The limitation of LiDAR (Light Detection And Ranging) sensor causes the general sparsity of produced depth measurement. However, the sparse representation of the world is insufficient for applications such as 3D reconstruction. Thus, depth completion is an important computer vision task in which a synchronized RGB image is commonly available. In this paper, we propose a deep neural network to tackle this image guided depth completion problem. By revisiting the sparsity invariant convolution and revealing how it can be used in a novel approach, we propose three mask aware operations to process, downscale, and fuse sparse features. These operations explicitly consider the observation mask of its corresponding feature map. In addition, the structure of this network follows a novel scheme in which data from image and depth domain are processed by these proposed operations independently. Our proposed model achieves state-of-the-art performance on the KITTI depth completion benchmark. Furthermore, it presents a strong robustness for bearing input sparsity under different densities and patterns.

Non-Learning Stereo-Aided Depth Completion Under Mis-Projection via Selective Stereo Matching

“…Thus, we filter out occlusion boundaries that are detected on the ground by Γ SSM in Eq. (18). Finally, by denoting B-ADT at pixel x ∈ Ω 1 as G x , we set G x based on the boundary conditions and the ground mask as follows:.…”

DVMN: Dense Validity Mask Network for Depth Completion

“…Similarly, binary masks have been used to filter invalid values [4]. The MA-bottleneck block of Yan et al [5] combines SI-convolution with a residual bottleneck block [6], as this block aids in gradient propagation, reduces parameters and saves computational costs. Furthermore they propose the MA-fusion module, effectively combining features at decoder skip connections while reintroducing binary validity information.…”

“…Multi-scale information improves the capability of networks to overcome differently sized or deformed input. Various networks integrate Spatial Pyramid Pooling (SPP) [22] for depth completion [5], [16], [18], [23]. Atrous Spatial Pyramid Pooling (ASPP) [24] has been studied at the end of an encoder [16] or within residual blocks [13].…”

“…Furthermore, we propose the SI-Residual Bottleneck, inspired by He et al [6] and Yan et al [5]. However, Yan et al [5] limit mask propagation to down-sampling layers. Sequential use of the same validity mask zeroes valid output at the next layer.…”

“…Each stage expands the channels by C•stage using a SI-Residual Bottleneck, then down-samples using strided SI-convolution, followed by five subsequent SI-Residual Bottlenecks. This results in an encoder output of 128 feature maps at 1 16 height and The output of both encoders are fused with added multiscale SPP context analogous to Yan et al [5], reducing features at a ratio of 2:1.…”