An Embodied Multi-Sensor Fusion Approach to Visual Motion Estimation Using Unsupervised Deep Networks

Abstract

Aimed at improving size, weight, and power (SWaP)-constrained robotic vision-aided state estimation, we describe our unsupervised, deep convolutional-deconvolutional sensor fusion network, Multi-Hypothesis DeepEfference (MHDE). MHDE learns to intelligently combine noisy heterogeneous sensor data to predict several probable hypotheses for the dense, pixel-level correspondence between a source image and an unseen target image. We show how our multi-hypothesis formulation provides increased robustness against dynamic, heteroscedastic sensor and motion noise by computing hypothesis image mappings and predictions at 76–357 Hz depending on the number of hypotheses being generated. MHDE fuses noisy, heterogeneous sensory inputs using two parallel, inter-connected architectural pathways and n (1–20 in this work) multi-hypothesis generating sub-pathways to produce n global correspondence estimates between a source and a target image. We evaluated MHDE on the KITTI Odometry dataset and benchmarked it against the vision-only DeepMatching and Deformable Spatial Pyramids algorithms and were able to demonstrate a significant runtime decrease and a performance increase compared to the next-best performing method.

Unsupervised Deep Visual-Inertial Odometry with Online Error Correction for RGB-D Imagery

“…In AirSim, the eight hypothesis networks generally outperformed the four hypothesis networks (as seen in Table 7) but on KITTI, the opposite was true. Previous results from [48] suggest a positive correlation between noise in the input data and benefits of additional hypotheses which may also translate to variance in the input data. As the MAV Air-Sim trajectories typically exhibited more varied motion compared to KITTI trajectories, this may explain why the eight-hypothesis networks seem to have been better utilized on AirSim compared to KITTI.…”

“…The final level of VIOLearner employs multi-hypothesis pathways similar to [48] where several possible hypotheses for the reconstructions of a target image (and the associated transformationsû m , m 2 M which generated those reconstructions) are computed in parallel. The lowest error hypothesis reconstruction is chosen during each network run and the corresponding affine matrixû mà which generated the winning reconstruction is output as the final network estimate of camera pose change between images I j and I jþ1 .…”

“…Error for this last multi-hypothesis level is computed according to a winner-take-all (WTA) euclidean loss rule (see [48] for more detail and justifications) Ir n;Ã À argmin k kIr n;k À I n jþ1 k 2 (5)…”