TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving

Chitta, Kashyap; Prakash, Aditya; Jaeger, Bernhard; Yu, Zehao; Renz, Katrin; Geiger, Andreas

doi:10.48550/arxiv.2205.15997

Cited by 4 publications

(7 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To comprehensively address the multiperspective requirements of our model and the following framework mention in Section 3.4, we use three typical end-to-end autonomous driving model: neat (Chitta et al, 2021), transfuser (Chitta et al, 2022) and LAV (Chen and Krähenbühl, 2022) as the standard navigation model in our evaluation. Neat presents an overall approach based on a multilayer perceptron neural attention field to conduct fully endto-end navigation, which first takes image patches from front, left, right three perspectives in 180-degree range, combined with velocity features as inputs and then summed and fed into a transformer with attention mechanism for final navigation tasks.…”

Section: Navigation Robustness Enhancement Evaluationmentioning

confidence: 99%

“…As mentioned above, we select three multiperspective navigation methods neat (Chitta et al, 2021), transfuser (Chitta et al, 2022) and LAV (Chen and Krähenbühl, 2022) combined with our reconstruction model to fully evaluate the augmentation of the robustness from our model in autonomous driving with various driving scenarios. In our evaluation, we consider two different routes town02 and town05 in CARLA as our test scenarios under various disturbances and attacks.…”

Section: Navigation Robustness Comparisonmentioning

confidence: 99%

“…Navigation results with our information reconstruction model on CARLA simulator under disturbances and attacksNeat(Chitta et al, 2021) Transfuser(Chitta et al, 2022) LAV(Chen and Krähenbühl, 2022) DS : IPK ; RC : DS : IPK ; RC : DS : IPK ; RC : DS : IPK ; RC : DS : IPK ; RC : DS : IPK ; Clean 94.398 86.398 0.283 94.381 79.172 0.317 94.453 91.231 0.000 92.110 67.327 0.212 89.699 89.699 0.000 93.001 91.001 0.000 All anomaly (without rec) 20.497 18.712 1.283 8.527 5.384 N/A 46.666 29.742 1.968 12.325 3.415 4.023 41.737 39.841 0.298 53.651 48.008 0.301 Only disturbances (with rec) 87.420 83.103 1.129 81.294 73.256 3.837 87.822 83.257 1.893 78.089 64.461 2.441 83.473 83.473 0.000 87.980 86.459 0.000 Only FGSM attacks (with rec) 87.654 81.580 0.688 80.104 72.486 3.142 84.070 82.236 0.460 75.340 62.815 1.563 89.882 89.882 0.000 94.376 90.861 0.000 All anomaly (with rec) 88.716 84.716 0.141 82.768 71.806 2.774 87.528 78.085 0.469 77.279 69.690 1.706 85.454 85.454 0.000 89.269 84.744 0.000…”

mentioning

confidence: 99%

See 2 more Smart Citations

Graph neural network based abnormal perception information reconstruction and robust autonomous navigation

Zhang,

Liu,

Miao

et al. 2024

RIA

View full text Add to dashboard Cite

Purpose This paper aims to develop a robust navigation enhancement framework to handle one of the most urgent needs for real applications of autonomous vehicles nowadays, as these corner cases act as the most commonly occurred risks in potential self-driving accidents. Design/methodology/approach In this paper, the main idea is to fully exploit the consistent features among spatio-temporal data and thus detect the anomalies and build residual channels to reconstruct the abnormal information. The authors first develop an anomaly detection algorithm, then followed by a corresponding disturbed information reconstruction network which has strong robustness to address both the nature disturbances and external attacks. Finally, the authors introduce a fully end-to-end resilient navigation performance enhancement framework to improve the driving performance of existing self-driving models under attacks and disturbances. Findings Comparison results on CARLA platform and real experiments demonstrate strong resilience of the authors’ approach which enhances the navigation performance under disturbances and attacks. Originality/value Reliable and resilient navigation performance under various nature disturbances and even external attacks is one of the most urgent needs for real applications of autonomous vehicles nowadays, as these corner cases act as the most commonly occurred risks in potential self-driving accidents. The information reconstruction approach provides a resilient navigation performance enhancement method for existing self-driving models.

show abstract

Section: Navigation Robustness Enhancement Evaluationmentioning

confidence: 99%

Section: Navigation Robustness Comparisonmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Graph neural network based abnormal perception information reconstruction and robust autonomous navigation

Zhang,

Liu,

Miao

et al. 2024

RIA

View full text Add to dashboard Cite

show abstract

“…To satisfy this time-varying power balance constraint, the safety layer provides a feasible solution space, which enables the RL agent to explore safely in it. Such a safety layer is common in many other similar tasks with high-security requirements [31,32].…”

Section: Das Model-free Rl Gridzeromentioning

confidence: 99%

Real-time scheduling of renewable power systems through planning-based reinforcement learning

Liu¹,

Wang²,

Ye³

et al. 2023

Preprint

View full text Add to dashboard Cite

The growing renewable energy sources have posed significant challenges to traditional power scheduling. It is difficult for operators to obtain accurate day-ahead forecasts of renewable generation, thereby requiring the future scheduling system to make real-time scheduling decisions aligning with ultra-short-term forecasts. Restricted by the computation speed, traditional optimization-based methods can not solve this problem. Recent developments in reinforcement learning (RL) have demonstrated the potential to solve this challenge. However, the existing RL methods are inadequate in terms of constraint complexity, algorithm performance, and environment fidelity. We are the first to propose a systematic solution

show abstract

“…LAV [45] trains the driving agent with the dataset collected from all the vehicles that it observes. Transfuser [8,46] is an imitation learning method where the agent uses transformer to fuse information from the front camera image and LiDAR information. The entries "Latent Transfuser" and "Trans-fuser+" are variants of Transfuser.…”

Section: Route Completionmentioning

confidence: 99%

Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

Shao¹,

Wang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Large-scale deployment of autonomous vehicles has been continually delayed due to safety concerns. On the one hand, comprehensive scene understanding is indispensable, a lack of which would result in vulnerability to rare but complex traffic situations, such as the sudden emergence of unknown objects. However, reasoning from a global context requires access to sensors of multiple types and adequate fusion of multi-modal sensor signals, which is difficult to achieve. On the other hand, the lack of interpretability in learning models also hampers the safety with unverifiable failure causes. In this paper, we propose a safety-enhanced autonomous driving framework, named Interpretable Sensor Fusion Transformer (InterFuser), to fully process and fuse information from multimodal multi-view sensors for achieving comprehensive scene understanding and adversarial event detection. Besides, intermediate interpretable features are generated from our framework, which provide more semantics and are exploited to better constrain actions to be within the safe sets. We conducted extensive experiments on CARLA benchmarks, where our model outperforms prior methods, ranking the first on the public CARLA Leaderboard. Our code will be made available at https://github.com/opendilab/InterFuser.

show abstract

TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving

Cited by 4 publications

References 49 publications

Graph neural network based abnormal perception information reconstruction and robust autonomous navigation

Graph neural network based abnormal perception information reconstruction and robust autonomous navigation

Real-time scheduling of renewable power systems through planning-based reinforcement learning

Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

Contact Info

Product

Resources

About