Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling

Zou, Yuliang; Ji, Pan; Tran, Quoc-Huy; Huang, Jia-Bin; Chandraker, Manmohan

doi:10.1007/978-3-030-58568-6_42

Cited by 69 publications

(38 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…11 and quantitative results are shown in Table IV. As can be seen, the proposed method significantly outperforms all the other learning-based VO methods, which includes a long-term model trained with 97 frames [49] and the algorithms [50] trained online on a test set. Compared with a pure geometry-based algorithm, such as ORB-SLAM [3] or DSO [4], the proposed model also achieves remarkable advantages in sequence 09.…”

Section: Odometry Evaluationmentioning

confidence: 90%

“…In contrast, to avoid the need for annotated data, self-supervised VO has been developed using the SfM pipeline. These methods accept continuous image input and infer VO via a CNN [7], [8] or LSTM [49], [50]. To combine the advantages of geometrybased and deep-learning methods, several works [10], [9] have separately learned the various components (e.g., optical flow, depth, and VO) of the entire system; others [11] have tried to combine them and train them jointly.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Generalized Visual Odometry Using Position-Aware Optical Flow and Geometric Bundle Adjustment

Cao¹,

Zhang²,

Luo³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, an essential problem of robust visual odometry (VO) is approached by incorporating geometry-based methods into deep-learning architecture in a self-supervised manner. Generally, pure geometry-based algorithms are not as robust as deep learning in feature-point extraction and matching, but perform well in ego-motion estimation because of their well-established geometric theory. In this work, a novel optical flow network (PANet) built on a position-aware mechanism is proposed first. Then, a novel system that jointly estimates depth, optical flow, and ego-motion without a typical network to learning ego-motion is proposed. The key component of the proposed system is an improved bundle adjustment module containing multiple sampling, initialization of ego-motion, dynamic damping factor adjustment, and Jacobi matrix weighting. In addition, a novel relative photometric loss function is advanced to improve the depth estimation accuracy. The experiments show that the proposed system not only outperforms other state-of-the-art methods in terms of depth, flow, and VO estimation among self-supervised learning-based methods on KITTI dataset, but also significantly improves robustness compared with geometrybased, learning-based and hybrid VO systems. Further experiments show that our model achieves outstanding generalization ability and performance in challenging indoor (TMU-RGBD) and outdoor (KAIST) scenes.

show abstract

Section: Odometry Evaluationmentioning

confidence: 90%

Section: Related Workmentioning

confidence: 99%

Learning Generalized Visual Odometry Using Position-Aware Optical Flow and Geometric Bundle Adjustment

Cao¹,

Zhang²,

Luo³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To overcome the scale inconsistency of SSM-VO, Bian et al [1] propose a geometry consistency loss to constrain the learning process, obtaining more accurate and scale-consistent results. Recently, inspired by geometric methods, Zou et al [32] propose a new self-supervised framework with a two-layer ConvLSTM [26], which maintains local and global consistency. Although these methods obtain some positive results, their self-supervised loss function relies on the consistency loss.…”

Section: B Self-supervised Vo Methodsmentioning

confidence: 99%

“…We select MonoDepth2 [8] and SC-Depth [2] as our baseline SSM-VO systems. SC-Depth purportedly outperforms all previous monocular alternatives except the most recent approach [32]. However, [32] is not open-sourced, so we consider SC-Depth as the best open-sourced SSM-VO system.…”

Section: A Experiments Setupmentioning

confidence: 99%

MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints

Wang¹,

Manocha²

2021

Preprint

View full text Add to dashboard Cite

We present a novel self-supervised algorithm named MotionHint for monocular visual odometry (VO) that takes motion constraints into account. A key aspect of our approach is to use an appropriate motion model that can help existing self-supervised monocular VO (SSM-VO) algorithms to overcome issues related to the local minima within their self-supervised loss functions. The motion model is expressed with a neural network named PPnet. It is trained to coarsely predict the next pose of the camera and the uncertainty of this prediction. Our self-supervised approach combines the original loss and the motion loss, which is the weighted difference between the prediction and the generated ego-motion. Taking two existing SSM-VO systems as our baseline, we evaluate our MotionHint algorithm on the standard KITTI benchmark. Experimental results show that our MotionHint algorithm can be easily applied to existing open-sourced state-of-the-art SSM-VO systems to greatly improve the performance by reducing the resulting ATE by up to 28.73%.

show abstract

“…Some focused on solving specific issues that often occur in training and at test time, e.g., the scale inconsistency issue [17,36,39] and the mixed information brought in by independent motion or static frames [2,11,20]. Better network architectures have been proposed [12,23,44,50]. Additional training objectives that extend the photometric constraint [42,45] or introduce geometric constraints [2,19,26,41,42] have been found useful.…”

Section: Monocular Depth Estimationmentioning

confidence: 99%

ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction

Zhang

Frahm

Ehrenstein

et al. 2021

Preprint

View full text Add to dashboard Cite

One of the key elements of reconstructing a 3D mesh from a monocular video is generating every frame's depth map. However, in the application of colonoscopy video reconstruction, producing good-quality depth estimation is challenging. Neural networks can be easily fooled by photometric distractions or fail to capture the complex shape of the colon surface, predicting defective shapes that result in broken meshes. Aiming to fundamentally improve the depth estimation quality for colonoscopy 3D reconstruction, in this work we have designed a set of training losses to deal with the special challenges of colonoscopy data. For better training, a set of geometric consistency objectives was developed, using both depth and surface normal information. Also, the classic photometric loss was extended with feature matching to compensate for illumination noise. With the training losses powerful enough, our self-supervised framework named ColDE is able to produce better depth maps of colonoscopy data as compared to the previous work utilizing prior depth knowledge. Used in reconstruction, our network is able to reconstruct good-quality colon meshes in real-time without any post-processing, making it the first to be clinically applicable.

show abstract

Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling

Cited by 69 publications

References 41 publications

Learning Generalized Visual Odometry Using Position-Aware Optical Flow and Geometric Bundle Adjustment

Learning Generalized Visual Odometry Using Position-Aware Optical Flow and Geometric Bundle Adjustment

MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints

ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction

Contact Info

Product

Resources

About