Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving

Huang, Zhiyu; Mo, Xiaoyu; Lv, Chen

doi:10.1109/icra46639.2022.9812060

Cited by 77 publications

(49 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-modal fusion is a method of combining data collected from different modalities to achieve more accurate results [29]. It is widely used in various fields from affective computing [30] to autonomous driving [31]. Recent studies have demonstrated that by using a combination of visual, vocal, and textual data, it is possible to more accurately identify psychological patterns from multiple perspectives [30], [32].…”

Section: B Multi-modal Fusionmentioning

confidence: 99%

CloudNet: A LiDAR-Based Face Anti-Spoofing Model That Is Robust Against Light Variation

Kim

Gwak

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Face anti-spoofing (FAS) is a technology that protects face recognition systems from presentation attacks. The current challenge faced by FAS studies is the difficulty in creating a generalized light variation model. This is because face data are sensitive to light domain. FAS models using only red green blue (RGB) images suffer from poor performance when the training and test datasets have different light variations. To overcome this problem, this study focuses on light detection and ranging (LiDAR) sensors. LiDAR is a time-of-flight depth sensor that is included in the latest mobile devices. It is negligibly affected by light and provides 3D coordinate and depth information of the target. Thus, a model that is resistant to light variations and exhibiting excellent performance can be created. For the experiment, datasets collected with a LiDAR camera are built and CloudNet architectures for RGB, point clouds, and depth are designed. Three protocols are used to confirm the performance of the model according to variations in the light domain. Experimental results indicate that for protocols 2 and 3, CloudNet error rates increase by 0.1340 and 0.1528, whereas the error rates of the RGB model increase by 0.3951 and 0.4111, respectively, as compared with protocol 1. These results demonstrate that the LiDAR-based FAS model with CloudNet has a more generalized performance compared with the RGB model.

show abstract

Section: B Multi-modal Fusionmentioning

confidence: 99%

CloudNet: A LiDAR-Based Face Anti-Spoofing Model That Is Robust Against Light Variation

Kim

Gwak

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Motion prediction is a thread of research that aims to predict long-term future motion trajectories of traffic participants based on their historical dynamic states and optionally the map information. With the introduction of Transformers or GNNs, recent motion prediction networks have gained the ability to effectively handle a heterogeneous mix of traffic entities, e.g., road polylines, traffic light state, and a dynamic set of agents, and achieved unprecedented prediction accuracy [14], [15], [26], [27]. However, most of the existing motion prediction models only focus on improving the prediction accuracy (i.e., position error), ignoring the applicability to the downstream planning task.…”

Section: B Motion Predictionmentioning

confidence: 99%

“…Scene Encoder. The scene encoder is based on Transformer networks, following the structure of our previous works [14], [33]. For each surrounding agent i, the input data consists of its historical states x i −T h :0 and local map polylines I i (i.e., a list of waypoints of nearby routes).…”

Section: B Prediction Modelmentioning

confidence: 99%

“…On the other hand, abundant research has been conducted on motion prediction models that directly forecast the agent's states several seconds into the future based on historical observations and corresponding maps. Many state-of-the-art prediction models use Transformers or graph neural networks (GNNs) [13], [14], [15] to encode agent-agent and agent-map interactions have achieved outstanding prediction accuracy. Moreover, the future states of agents can be obtained via one forward pass, which could significantly accelerate the decision-making process and reduce latency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Human-Like Decision Making and Control for Autonomous Driving

Hang¹,

Lv²,

Chen³

2022

View full text Add to dashboard Cite

Predicting the behaviors of other road users is crucial to safe and intelligent decision-making for autonomous vehicles (AVs). However, most motion prediction models ignore the influence of the AV's actions and the planning module has to treat other agents as unalterable moving obstacles. To address this problem, this paper proposes an interactionaware motion prediction model that is able to predict other agents' future trajectories according to the ego agent's future plan, i.e., their reactions to the ego's actions. Specifically, we employ Transformers to effectively encode the driving scene and incorporate the AV's future plan in decoding the predicted trajectories. To train the model to accurately predict the reactions of other agents, we develop an online learning framework, where the ego agent explores the environment and collects other agents' reactions to itself. We validate the decision-making and learning framework in three highly interactive simulated driving scenarios. The results reveal that our decision-making method significantly outperforms the reinforcement learning methods in terms of data efficiency and performance. We also find that using the interaction-aware model can bring better performance than the non-interaction-aware model and the exploration process helps improve the success rate in testing.

show abstract

“…It has been experimentally validated in urban settings involving complex scenarios using the CARLA urban driving simulator. Furthermore, Huang et al [7] introduced a neural prediction framework utilizing the Transformer structure, with a multi-modal attention mechanism for representing social interactions between agents and predicting multiple trajectories for autonomous driving. In our approach, we do not train a transformer-based model from scratch, as this would be too expensive to do for each application.…”

Section: A Multimodal Transformersmentioning

confidence: 99%

Human-in-the-loop online multi-agent approach to increase trustworthiness in ML models through trust scores and data augmentation

Bravo-Rocca

Liu

Guitart

et al. 2022

2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)

View full text Add to dashboard Cite

Increasing a ML model accuracy is not enough, we must also increase its trustworthiness. This is an important step for building resilient AI systems for safety-critical applications such as automotive, finance, and healthcare. For that purpose, we propose a multi-agent system that combines both machine and human agents. In this system, a checker agent calculates a trust score of each instance (which penalizes overconfidence in predictions) using an agreement-based method and ranks it; then an improver agent filters the anomalous instances based on a human rule-based procedure (which is considered safe), gets the human labels, applies geometric data augmentation, and retrains with the augmented data using transfer learning. We evaluate the system on corrupted versions of the MNIST and FashionMNIST datasets. We get an improvement in accuracy and trust score with just few additional labels compared to a baseline approach.

show abstract

Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving

Cited by 77 publications

References 18 publications

CloudNet: A LiDAR-Based Face Anti-Spoofing Model That Is Robust Against Light Variation

CloudNet: A LiDAR-Based Face Anti-Spoofing Model That Is Robust Against Light Variation

Human-Like Decision Making and Control for Autonomous Driving

Human-in-the-loop online multi-agent approach to increase trustworthiness in ML models through trust scores and data augmentation

Contact Info

Product

Resources

About