Fusion of Stereo and Still Monocular Depth Estimates in a Self-Supervised Learning Context

Martins, Diogo; Hecke, Kevin van; Croon, Guido de

doi:10.1109/icra.2018.8461116

Cited by 22 publications

(12 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 7 reports experiment from sequence 2011 09 26 0011 of the KITTI raw dataset [6]. We compare our framework with fusion strategies proposed by Martins et al [18] and Marin et al [17], combining outputs by the stereo networks respectively with monocular estimates (using the network by Guo et al [9]) and Lidar, reporting the ideal result as in [17]. Ground-truth labels for evaluation are provided by [39].…”

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

See 1 more Smart Citation

Guided Stereo Matching

Poggi

Pallotti

Tosi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

93.21% 4.36% (a) (b) (c) Figure 1. Guided stereo matching. (a) Challenging, reference image from KITTI 2015 [20] and disparity maps estimated by (b) iResNet [14] trained on synthetic data [19], or (c) guided by sparse depth measurements (5% density). Error rate (> 3) superimposed on each map. AbstractStereo is a prominent technique to infer dense depth maps from images, and deep learning further pushed forward the state-of-the-art, making end-to-end architectures unrivaled when enough data is available for training. However, deep networks suffer from significant drops in accuracy when dealing with new environments. Therefore, in this paper, we introduce Guided Stereo Matching, a novel paradigm leveraging a small amount of sparse, yet reliable depth measurements retrieved from an external source enabling to ameliorate this weakness. The additional sparse cues required by our method can be obtained with any strategy (e.g., a LiDAR) and used to enhance features linked to corresponding disparity hypotheses. Our formulation is general and fully differentiable, thus enabling to exploit the additional sparse inputs in pre-trained deep stereo networks as well as for training a new instance from scratch. Extensive experiments on three standard datasets and two stateof-the-art deep architectures show that even with a small set of sparse input cues, i) the proposed paradigm enables significant improvements to pre-trained networks. Moreover, ii) training from scratch notably increases accuracy and robustness to domain shifts. Finally, iii) it is suited and effective even with traditional stereo algorithms such as SGM.

show abstract

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

“…<2% avg. All NoG All NoG iResNet [14] 18.42 18.37 1.28 1.28 iResNet+Martins et al [18] 18 is not limited to pixels with associated Lidar measurement in contrast to fusion techniques [17].…”

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

Guided Stereo Matching

Poggi

Pallotti

Tosi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…The image is synthesized from the network outputs, following the traditional Structure-frommotion procedure. Extra constraint and additional information have been introduced to improve the performance, like the temporal depth consistency [22], the stereo matching [23] and the semantic information [34]. Godard et al [11] achieved a significant improvement by compensating for image occlusion.…”

Section: Supervised Depth Estimationmentioning

confidence: 99%

Self-supervised Object Motion and Depth Estimation from Video

Dai

Patil

Hecker

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

We present a self-supervised learning framework to estimate the individual object motion and monocular depth from video. We model the object motion as a 6 degree-offreedom rigid-body transformation. The instance segmentation mask is leveraged to introduce the information of object. Compared with methods which predict dense optical flow map to model the motion, our approach significantly reduces the number of values to be estimated. Our system eliminates the scale ambiguity of motion prediction through imposing a novel geometric constraint loss term. Experiments on KITTI driving dataset demonstrate our system is capable to capture the object motion without external annotation. Our system outperforms previous self-supervised approaches in terms of 3D scene flow prediction, and contribute to the disparity prediction in dynamic area.

show abstract

“…Similarly, Fácil et al [58] fused CNN-based single-view and multi-view depth to improve the depth of low-parallax image sequences. Martins et al [59] have demonstrated that the stereo depth leads to higher performance with the monocular estimated depth fusion.…”

Section: B Depth Estimationmentioning

confidence: 99%

Image-Based Rendering for Large-Scale Outdoor Scenes With Fusion of Monocular and Multi-View Stereo Depth

Liu

Zhang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Image-based rendering (IBR) attempts to synthesize novel views using a set of observed images. Some IBR approaches (such as light fields) have yielded impressive high-quality results on smallscale scenes with dense photo capture. However, available wide-baseline IBR methods are still restricted by the low geometric accuracy and completeness of multi-view stereo (MVS) reconstruction on low-textured and non-Lambertian surfaces. The issues become more significant in large-scale outdoor scenes due to challenging scene content, e.g., buildings, trees, and sky. To address these problems, we present a novel IBR algorithm that consists of two key components. First, we propose a novel depth refinement method that combines MVS depth maps with monocular depth maps predicted via deep learning. A lookup table remap is proposed for converting the scale of the monocular depths to be consistent with the scale of the MVS depths. Then, the rescaled monocular depth is used as the constraint in the minimum spanning tree (MST)based nonlocal filter to refine the per-view MVS depth. Second, we present an efficient shape-preserving warping algorithm that uses superpixels to generate the warped images and blend expected novel views of scenes. The proposed method has been evaluated on public MVS and view synthesis datasets, as well as newly captured large-scale outdoor datasets. In comparison with state-of-the-art methods, the experimental results demonstrated that the proposed method can obtain more complete and reliable depth maps for the challenging large-scale outdoor scenes, thereby resulting in more promising novel view synthesis. INDEX TERMS Image-based rendering, multi-view stereo, monocular depth estimation, view synthesis, outdoor scenes.

show abstract

Fusion of Stereo and Still Monocular Depth Estimates in a Self-Supervised Learning Context

Cited by 22 publications

References 22 publications

Guided Stereo Matching

Guided Stereo Matching

Self-supervised Object Motion and Depth Estimation from Video

Image-Based Rendering for Large-Scale Outdoor Scenes With Fusion of Monocular and Multi-View Stereo Depth

Contact Info

Product

Resources

About