RigNet: Repetitive Image Guided Network for Depth Completion

Abstract

Depth completion aims to recover dense depth maps from sparse ones, where color images are often used to facilitate this task. Recent depth methods primarily focus on image guided learning frameworks. However, blurry guidance in the image and unclear structure in the depth still impede their performance. To tackle these challenges, we explore an efficient repetitive design in our image guided network to gradually and sufficiently recover depth values. Specifically, the efficient repetition is embodied in both the image guidance branch and depth generation branch. In the former branch, we design a dense repetitive hourglass network to extract discriminative image features of complex environments, which can provide powerful contextual instruction for depth prediction. In the latter branch, we introduce a repetitive guidance module based on dynamic convolution, in which an efficient convolution factorization is proposed to reduce the complexity while modeling high-frequency structures progressively. Extensive experiments indicate that our approach achieves superior or competitive results on KITTI, VKITTI, NYUv2, 3D60, and Matterport3D datasets.

AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization

“…SfM-Learner (Zhou et al 2017) is a seminal work that proposed to jointly predict scene depth and relative camera poses. Follow-up works enhanced SfM-Learner by decomposing depth scale (Wang et al 2021b;Yan et al 2023), introducing powerful neural networks (Guizilini et al 2020;Lyu et al 2021;Guizilini et al 2022), and applying iterative refinement (Bangunharcana, Magd, and Kim 2023). Furthermore, MonoDepth2 (Godard et al 2019) proposed a minimum reprojection loss to handle occlusions, and some works addressed the dynamic object problem by compensating and masking pixels within dynamic areas using optical flow (Zou, Luo, and Huang 2018;Ranjan et al 2019) and pretrained segmentation models (Gordon et al 2019).…”

SPNet: Structure preserving network for depth completion

“…Deep neural networks greatly promote the development of depth completion task. At present, the related works of depth completion can be roughly divided into three main categories: single-branch-based methods [4,18,19,[35][36][37][38], two-branch-based methods [16,17,20,22,[39][40][41][42][43] and multiple-branch-based methods [3,25,44]. The single-branch-based methods use only one encoder-decoder network to complete depth maps.…”

“…For example, Tang et al [16] used two independent encoder-decoder network to extract the RGB and depth features respectively, and then designed a guided convolution module to fuse the decoder features from RGB branch with the encoder features from depth branch. Besides, some multiple-branch-based works [3,44] employed three hourglass networks to extract more useful information. Yan et al [44] indicated that the increasing number of hourglass networks could improve the depth completion performance.…”

“…Besides, some multiple-branch-based works [3,44] employed three hourglass networks to extract more useful information. Yan et al [44] indicated that the increasing number of hourglass networks could improve the depth completion performance. Although these existing methods do improve the overall quality of depth maps, further research is still necessary to better preserve depth structures.…”

RGB-Depth Fusion GAN for Indoor Depth Completion

“…Ma and Karaman [10] used a CNN encoder-decoder to predict the full-resolution depth image from a set of depth samples and RGB images. On this basis, several methods [11], [12], [19], [31], [32] incorporating additional representations or auxiliary outputs have been proposed. Qiu et al [11] produced dense depth using the surface normal as the intermediate representation.…”