Selective Sensor Fusion for Neural Visual-Inertial Odometry

Abstract

Deep learning approaches for Visual-Inertial Odometry (VIO) have proven successful, but they rarely focus on incorporating robust fusion strategies for dealing with imperfect input sensory data. We propose a novel end-to-end selective sensor fusion framework for monocular VIO, which fuses monocular images and inertial measurements in order to estimate the trajectory whilst improving robustness to real-life issues, such as missing and corrupted data or bad sensor synchronization. In particular, we propose two fusion modalities based on different masking strategies: deterministic soft fusion and stochastic hard fusion, and we compare with previously proposed direct fusion baselines. During testing, the network is able to selectively process the features of the available sensor modalities and produce a trajectory at scale. We present a thorough investigation on the performances on three public autonomous driving, Micro Aerial Vehicle (MAV) and hand-held VIO datasets. The results demonstrate the effectiveness of the fusion strategies, which offer better performances compared to direct fusion, particularly in presence of corrupted data. In addition, we study the interpretability of the fusion networks by visualising the masking layers in different scenarios and with varying data corruption, revealing interesting correlations between the fusion networks and imperfect sensory input data.

Unsupervised Monocular Visual-inertial Odometry Network

“…VINet firstly tackled VIO in a supervised manner. Chen et al [Chen et al, 2019] exploited two masking strategies for visual-inertial sensor fusion.…”

“…With the development of CNN and RNN, various learningbased VO or VIO methods have been proposed. Although many supervised methods [Wang et al, 2018, Chen et al, 2019 have been revealed more competitive than traditional methods, the demands of a large number of labeled data, i.e., the ground truth poses acquired from high-precision devices, limit the application of the technology. Self-supervised methods [Shamwell et al, 2018, Han et al, 2019 release the pressure of collecting large quantities of ground truth, but they still require other expensive data, e.g., depth map, which degrades the flexibility.…”

Adaptive Multimodal Fusion for Facial Action Units Recognition

“…One of the key steps in multimodal machine learning is the multimodal fusion, with the aim at integrating features of multiple modalities for enabling more accurate and robust performance. Three types of fusion strategies (i.e., early, late, and hybrid fusion) are the commonly used techniques for multimodal feature fusion [2][12] [5]. Our proposed adaptive feature fusion strategy is particularly related to late fusion with a focus on the selection mechanism in order to choose the most relevant feature representations from different modalities, meanwhile it can avoid useless or misleading information.…”

“…, ∈ { , , }, and ∈ [0, 1] representing the score for individual feature extracted from different modalities. Instead of re-weighting each feature by the corresponding score, we apply a stochastic fusion method [5] to select the feature from different modalities.…”

Robust Stereo Visual Inertial Navigation System Based on Multi-Stage Outlier Removal in Dynamic Environments

“…The optimization-based strategy is commonly more reliable than filtering-based and is more robust to outliers. Most recently, deep learning based-VINS [20][21][22] have employed the Convolutional Neural Networks (CNNs) and Long Short Term Memory (LSTM) networks [23] architecture into an end-to-end learning process [24] to handle vision and inertial data simultaneously. The evaluations are sufficient but less precise compared to the model-based systems.…”