MMFN: Multi-Modal-Fusion-Net for End-to-End Driving

Abstract

Inspired by the fact that humans use diverse sensory organs to perceive the world, sensors with different modalities are deployed in end-to-end driving to obtain the global context of the 3D scene. In previous works, camera and LiDAR inputs are fused through transformers for better driving performance. These inputs are normally further interpreted as high-level map information to assist navigation tasks. Nevertheless, extracting useful information from the complex map input is challenging, for redundant information may mislead the agent and negatively affect driving performance. We propose a novel approach to efficiently extract features from vectorized High-Definition (HD) maps and utilize them in the end-to-end driving tasks. In addition, we design a new expert to further enhance the model performance by considering multi-road rules. Experimental results prove that both of the proposed improvements enable our agent to achieve superior performance compared with other methods. The source code is released as an open-source package.• We use a multi-path consideration rule-based expert to

Robust and transferable end-to-end navigation against disturbances and external attacks: an adversarial training approach

“…Firstly, for sensor fusion framework, conditional imitation learning (CIL) (Codevilla et al, 2018) provides a structure of sensor data fusion policy, high-level commands (human intention or planned route) are used for channel switching. This framework is widely used in the most of the imitation learning-based models (Codevilla et al, 2018;Wang et al, 2019;Liang et al, 2018;Sobh et al, 2018;Ma et al, 2020;Huang et al, 2021;Zhang et al, 2022). Secondly, except the CIL structure used in most of researches, the high-level command can also be considered as an additional input observation to enhance the model ability of steering estimation (Codevilla et al, 2018;Wang et al, 2019).…”

“…However, this pipeline suffers from hand-tuned parameters, complex intermediate representations and other drawbacks. To alleviate the above challenges, a fully end-to-end fashion with imitation learning or reinforcement learning has been more and more popular in these years (Liang et al, 2018;Gao et al, 2017;Ma et al, 2020;Zhang et al, 2022), which exploits the potential of learning technologies and can achieve comparable performance with human drivers. Various learning-based end-to-end navigation approaches are presented in the literature (Codevilla et al, 2018;Liang et al, 2018;Gao et al, 2017;Ma et al, 2020;Huang et al, 2021), which directly learn the relationship between the surrounding environment and raw sensor observation data of the vehicle and its control policy, and have provided practical demonstrations of learning-based end-toend policy trained in real vehicles (Cui et al, 2022).…”

“…Nearly all the existing end-to-end navigation methods assume clean sensor data (Huang et al, 2021;Ma et al, 2020;Amini et al, 2019) and only a few works consider sensor noises and challenging weather conditions (Gao et al, 2017;Zhang et al, 2021;Lee et al, 2022;Liu et al, 2022). However, recent researches in the computer-vision community have clearly shown the fragility of deep-learning models in the presence of data distribution shifts (Koh et al, 2021), disturbances and attacks injected in the input data (Wu et al, 2018;Tsai et al, 2020;Zhang et al, 2022) and several demonstrations from the viewpoint of robotics community have also successfully illustrated how to perform attacks on the raw sensor data as well as the high-level planning system to influence the performance of self-driving systems (Zeng et al, 2018;Shen et al, 2021;Shu et al, 2021). Thus, the resilience issue is of great importance for real applications of any vehicle navigation model.…”