Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Prakash, Amit; Chitta, Kashyap; Geiger, Andreas

doi:10.1109/cvpr46437.2021.00700

Cited by 374 publications

(220 citation statements)

References 32 publications

Supporting

Mentioning

218

Contrasting

Order By: Relevance

“…Transfuser [35] and IA [42]. LBC is the state of the art on the NoCrash benchmark, and Transfuser is a very recent method utilizing sensor fusion.…”

Section: Methodsmentioning

confidence: 99%

Learning to drive from a world on rails

Chen¹,

Koltun²

2021

Preprint

View full text Add to dashboard Cite

We learn an interactive vision-based driving policy from pre-recorded driving logs via a model-based approach. A forward model of the world supervises a driving policy that predicts the outcome of any potential driving trajectory. To support learning from pre-recorded logs, we assume that the world is on rails, meaning neither the agent nor its actions influence the environment. This assumption greatly simplifies the learning problem, factorizing the dynamics into a nonreactive world model and a low-dimensional and compact forward model of the ego-vehicle. Our approach computes action-values for each training trajectory using a tabular dynamic-programming evaluation of the Bellman equations; these action-values in turn supervise the final vision-based driving policy. Despite the world-on-rails assumption, the final driving policy acts well in a dynamic and reactive world. Our method ranks first on the CARLA leaderboard, attaining a 25% higher driving score while using 40× less data. Our method is also an order of magnitude more sample-efficient than state-of-the-art model-free reinforcement learning techniques on navigational tasks in the ProcGen benchmark.

show abstract

“…Transfuser [35] and IA [42]. LBC is the state of the art on the NoCrash benchmark, and Transfuser is a very recent method utilizing sensor fusion.…”

Section: Methodsmentioning

confidence: 99%

Learning to drive from a world on rails

Chen¹,

Koltun²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Various multi-modal fusion methods have been proposed for extracting inter-modal relationship and fusing representations [23][24][25][26]. In particular, the use of multi-head self-attention (MHSA) based frameworks such as Transformer has recently been reported to be effective for linking different modalities [26,27]. MHSA is defined as the case where…”

Section: Related Studiesmentioning

confidence: 99%

“…Here, W Q i , W K i , and W V i are projection parameters, and d k are dimension of K. By applying different projections on the input for each head, it is possible to model the relationship between the input sequences from multiple perspectives. Although MHSA was originally proposed for natural language processing, now it has been reported that by using joint sequences from two modalities as input (e.g., video and word or image and light detection and ranging (LiDAR) image), it is possible to extract the relationship between them [26,27].…”

Section: Related Studiesmentioning

confidence: 99%

Multi-view and Multi-modal Event Detection Utilizing Transformer-based Multi-sensor fusion

Yasuda¹,

Ohishi²,

Saito³

et al. 2022

Preprint

View full text Add to dashboard Cite

We tackle a challenging task: multi-view and multi-modal event detection that detects events in a wide-range real environment by utilizing data from distributed cameras and microphones and their weak labels. In this task, distributed sensors are utilized complementarily to capture events that are difficult to capture with a single sensor, such as a series of actions of people moving in an intricate room, or communication between people located far apart in a room. For sensors to cooperate effectively in such a situation, the system should be able to exchange information among sensors and combines information that is useful for identifying events in a complementary manner. For such a mechanism, we propose a Transformer-based multisensor fusion (MultiTrans) which combines multi-sensor data on the basis of the relationships between features of different viewpoints and modalities. In the experiments using a dataset 1 newly collected for this task, our proposed method using MultiTrans improved the event detection performance and outperformed comparatives.

show abstract

“…[45] is an overview of learningbased SLAM and suggests that artificial intelligence (AI) and deep learning models can aid performance in cases with imperfect sensor measurement, environmental dynamics, or noise. Another application of deep learning in SLAM has been the improvement of sensor fusion in SLAM and end-to-end autonomous driving [46] [47] [48].…”

Section: Deep Learning Slammentioning

confidence: 99%

Autonomous Vehicles: Open-Source Technologies, Considerations, and Development

Oussama¹,

Singh²,

Pourreza³

2022

Preprint

View full text Add to dashboard Cite

Autonomous vehicles are the culmination of advances in many areas such as sensor technologies, artificial intelligence (AI), networking, and more. This paper will introduce the reader to the technologies that build autonomous vehicles. It will focus on open-source tools and libraries for autonomous vehicle development, making it cheaper and easier for developers and researchers to participate in the field. The topics covered are as follows. First, we will discuss the sensors used in autonomous vehicles and summarize their performance in different environments, costs, and unique features. Then we will cover Simultaneous Localization and Mapping (SLAM) and algorithms for each modality. Third, we will review popular open-source driving simulators, a cost-effective way to train machine learning models and test vehicle software performance. We will then highlight embedded operating systems and the security and development considerations when choosing one. After that, we will discuss Vehicle-to-Vehicle (V2V) and Internet-of-Vehicle (IoV) communication, which are areas that fuse networking technologies with autonomous vehicles to extend their functionality. We will then review the five levels of vehicle automation, commercial and opensource Advanced Driving Assistance Systems, and their features. Finally, we will touch on the major manufacturing and software companies involved in the field, their investments, and their partnerships. These topics will give the reader an understanding of the industry, its technologies, active research, and the tools available for developers to build autonomous vehicles.

show abstract

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Cited by 374 publications

References 32 publications

Learning to drive from a world on rails

Learning to drive from a world on rails

Multi-view and Multi-modal Event Detection Utilizing Transformer-based Multi-sensor fusion

Autonomous Vehicles: Open-Source Technologies, Considerations, and Development

Contact Info

Product

Resources

About