Multimodal End-to-End Autonomous Driving

Abstract

Autonomous vehicles (AVs) are key for the intelligent mobility of the future. A crucial component of an AV is the artificial intelligence (AI) able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception (e.g. object detection, semantic segmentation, depth estimation, tracking) and maneuver control (e.g. local path planing and control). On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals (e.g. the steering angle). The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., traditional LiDARs, or new solid state ones). Accordingly, this paper analyses if RGB and depth data, i.e. RGBD data, can actually act as complementary information in a multimodal end-to-end driving approach, producing a better AI driver. Using the CARLA simulator functionalities, its standard benchmark, and conditional imitation learning (CIL), we will show how, indeed, RGBD gives rise to more successful end-toend AI drivers. We will compare the use of RGBD information by means of early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings.

A Survey of End-to-End Driving: Architectures and Training Methods

“…The same software architecture was later also employed in 2007 DARPA urban challenge [18]. The modular nature of classical pipelines for autonomous driving has also been discussed in [2], [10], [19]- [24].…”

“…a misdetection [25]. More generally, modularity allows to reliably reason about how the system arrived at specific driving decisions [10], [24], [25].…”

“…The use of end-to-end optimization raises an issue of interpretability. With no intermediate outputs, it is much harder to trace the initial cause of an error as well as to explain why the model arrived at specific driving decisions [10], [24], [39], [40]. However, solutions exist to increase the interpretability of end-to-end driving models (discussed later in Section VII).…”

VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles

“…in scaling simulation engines to multiple sensor types for online perception learning remain. Because embodied agents benefit from rich perception [5], integrating multiple sensor modalities could facilitate adaptation to a wide variety of environmental conditions (e.g., combining LiDAR and camera feedback to stay on the road in low visibility lighting). There remains a need for unified, flexible, and open-source datadriven simulation engines to fuel the development of new algorithms for embodied agent learning and evaluation.…”

“…Driving Policy Learning: While policy learning for driving using real-world data is largely restricted to IL [31]- [35], learning in simulation allows for greater algorithmic flexibiltiy ranging from IL [5], [36], [37], to RL [1], [4], [38], [39], and GPL [6], [40]. Evaluation of trained policies in closed-loop simulation [5], [31], [36], [40]- [42] also presents benefits over open-loop evaluation [7], [32], [43]. Similarly, our work leverages simulation for edge-case training data generation, and closed-loop evaluation before deployment.…”