End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks

Abstract

This paper studies visual odometry (VO) from the perspective of deep learning. After tremendous efforts in the robotics and computer vision communities over the past few decades, state-of-the-art VO algorithms have demonstrated incredible performance. However, since the VO problem is typically formulated as a pure geometric problem, one of the key features still missing from current VO systems is the capability to automatically gain knowledge and improve performance through learning. In this paper, we investigate whether deep neural networks can be effective and beneficial to the VO problem. An end-to-end, sequence-to-sequence probabilistic visual odometry (ESP-VO) framework is proposed for the monocular VO based on deep recurrent convolutional neural networks. It is trained and deployed in an end-to-end manner, that is, directly inferring poses and uncertainties from a sequence of raw images (video) without adopting any modules from the conventional VO pipeline. It can not only automatically learn effective feature representation encapsulating geometric information through convolutional neural networks, but also implicitly model sequential dynamics and relation for VO using deep recurrent neural networks. Uncertainty is also derived along with the VO estimation without introducing much extra computation. Extensive experiments on several datasets representing driving, flying and walking scenarios show competitive performance of the proposed ESP-VO to the state-of-the-art methods, demonstrating a promising potential of the deep learning technique for VO and verifying that it can be a viable complement to current VO systems.

SGANVO: Unsupervised Deep Visual Odometry and Depth Estimation With Stacked Generative Adversarial Networks

“…To fine-tune the network, we used the original image size in computing the losses. For data prepossessing, different kinds of data augmentation methods were used to enhance the performance and mitigate possible over-fitting, such as image color augmentation [5], rotational data augmentation [24] and left-right pose estimation augmentation [4]. We increased the weight parameter of rotational data to achieve better performance because the magnitude of rotation is very small compared to that of translation.…”

“…The Discriminator training procedure is to make the W (x,x) convergent to the minimum. The above procedure is repeated [24], SfMlearner [5]; Stereo: UndeepVO [9], Undepthflow [11]), and one geometric based methods (ORB-SLAM). Our SGANVO, SfMLearner, and UndeepVO use images with 416×128.…”

“…The results are shown in Table III. We compare the results with ESP-VO [24], SfMLearner [5], ORB-SLAM (without loop closure), UndeepVO [9], and Undepthflow [11]. It can be seen that our SGANVO shows a better performance in the testing sequences (09, 10) with these state-of-the-art methods in terms of the ATE metric.…”

Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry

“…The Refining module ameliorates previous outputs by employing a spatialtemporal feature reorganization mechanism. 22,[31][32][33]. Due to the high dimensionality of depth maps, the number of frames is commonly limited to no more than 5.…”

DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints

“…Furthermore, all these VIO systems require careful parameter tuning procedures for the specific environment they have to work in. 1 In recent years, deep learning based VO has drawn significant attentions due to its potentials in learning capability and the robustness to camera parameters and challenging environments [8], [9]. These data-driven VO methods have successfully learned new feature representations from images that are used to further improve the motion estimation.…”