TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving

Chitta, Kashyap; Prakash, Aditya; Jaeger, Bernhard; Yu, Zehao; Renz, Katrin; Geiger, Andreas

doi:10.1109/tpami.2022.3200245

Cited by 137 publications

(35 citation statements)

References 100 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We measured the running time of our model on a single RTX 3090 GPU by averaging over all time steps of the evaluation route, as shown in Table 5 . Some of data in Table 5 are quoted from work (Chitta et al, 2022 ). Our model takes 35.3 ms per frame, an increase of 11.8 ms and 7.7 ms compared to LF(23.5 ms) and TF(27.6 ms), respectively.…”

Section: Methodsmentioning

confidence: 99%

Dual-flow network with attention for autonomous driving

Yang

Lei²,

Zhang³

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

We present a dual-flow network for autonomous driving using an attention mechanism. The model works as follows: (i) The perception network extracts red, blue, and green (RGB) images from the video at low speed as input and performs feature extraction of the images; (ii) The motion network obtains grayscale images from the video at high speed as the input and completes the extraction of object motion features; (iii) The perception and motion networks are fused using an attention mechanism at each feature layer to perform the waypoint prediction. The model was trained and tested using the CARLA simulator and enabled autonomous driving in complex urban environments, achieving a success rate of 74%, especially in the case of multiple dynamic objects.

show abstract

Section: Methodsmentioning

confidence: 99%

Dual-flow network with attention for autonomous driving

Yang

Lei²,

Zhang³

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…Most recently, a driving simulator built for behavioral research, DReyeVR [18], was presented -building upon the well-established CARLA driving simulator with added data collection, VR functionality, and hardware integration. DReyeVR's compatibility with CARLA makes VR user studies for driving democratized and thus opens doors for research which is compatible with many existing state-of-the-art autonomous driving methods, especially those built off Learning By Cheating [19], [20], [21].…”

Section: B Modeling Driving Behaviormentioning

confidence: 99%

Towards Driving Policies with Personality: Modeling Behavior and Style in Risky Scenarios via Data Collection in Virtual Reality

Zheng¹,

Poveda²,

Mullen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Autonomous driving research currently faces data sparsity in representation of risky scenarios. Such data is both difficult to obtain ethically in the real world, and unreliable to obtain via simulation. Recent advances in virtual reality (VR) driving simulators lower barriers to tackling this problem in simulation. We propose the first data collection framework for risky scenario driving data from real humans using VR, as well as accompanying numerical driving personality characterizations. We validate the resulting dataset with statistical analyses and model driving behavior with an eight-factor personality vector based on the Multi-dimensional Driving Style Inventory (MDSI). Our method, dataset, and analyses show that realistic driving personalities can be modeled without deep learning or large datasets to complement autonomous driving research.

show abstract

“…Moreover, our control method generalizes well to the viewpoint changes due to effective data augmentation. Furthermore, [14] and its extension [16] deploy an additional LiDAR sensor along with the desired goal position to predict the future waypoints of the vehicle at inference time. In our work, we only use RGB images to determine the position and orientation of the goal (target vehicle) at inference and do not require extra LiDAR sensor to be present in the vehicle setup.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, we use the CARLA driving simulator (version 0.9.11) [5] for our experiments. CARLA has been widely adapted for online evaluation in control algorithms such as [13], [14], [16] due to a wide range of sensors that can be attached to the ego-vehicle for both data collection and inference, as well as having maps with different terrains and routes. It also furnishes precise ground truth labels for various tasks such as semantic segmentation, optical flow, dense depth, etc.…”

Section: Experiments a Experimental Setupmentioning

confidence: 99%

Robust Autonomous Vehicle Pursuit Without Expert Steering Labels

Pan,

Zhou,

Gladkova

et al. 2023

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

In this work, we present a learning method for both lateral and longitudinal motion control of an ego-vehicle for the task of vehicle pursuit. The car being controlled does not have a pre-defined route, rather it reactively adapts to follow a target vehicle while maintaining a safety distance. To train our model, we do not rely on steering labels recorded from an expert driver, but effectively leverage a classical controller as an offline label generation tool. In addition, we account for the errors in the predicted control values, which can lead to a loss of tracking and catastrophic crashes of the controlled vehicle. To this end, we propose an effective data augmentation approach, which allows to train a network that is capable of handling different views of the target vehicle. During the pursuit, the target vehicle is firstly localized using a Convolutional Neural Network. The network takes a single RGB image along with cars' velocities and estimates target vehicle's pose with respect to the ego-vehicle. This information is then fed to a Multi-Layer Perceptron, which regresses the control commands for the ego-vehicle, namely throttle and steering angle. We extensively validate our approach using the CARLA simulator on a wide range of terrains. Our method demonstrates realtime performance, robustness to different scenarios including unseen trajectories and high route completion. Project page containing code and multimedia can be publicly accessed here: https://changyaozhou.github.io/Autonomous-Vehicle-Pursuit/.

show abstract

TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving

Cited by 137 publications

References 100 publications

Dual-flow network with attention for autonomous driving

Dual-flow network with attention for autonomous driving

Towards Driving Policies with Personality: Modeling Behavior and Style in Risky Scenarios via Data Collection in Virtual Reality

Robust Autonomous Vehicle Pursuit Without Expert Steering Labels

Contact Info

Product

Resources

About