Altruistic Maneuver Planning for Cooperative Autonomous Vehicles Using Multi-agent Advantage Actor-Critic

Toghi, Behrad; Valiente, Rodolfo; Sadigh, Dorsa; Pedarsani, Ramtin; Fallah, Yaser P.

doi:10.48550/arxiv.2107.05664

“…Finally, we compared state of the art architectures related to our approach [2], [3], [12], [49] in terms of safety and efficiency to validate H3. We trained the different architectures in the same situations and examined their performance under different levels of HVs behaviors.…”

Section: ) Safetymentioning

confidence: 99%

“…In a pursuit to alleviate the challenges of this co-existence and enable social navigation for AVs, existing works either rely on models of human behavior derived from pre-recorded driving datasets [10], [11] or defining social utilities that can enforce a cooperative behavior among AVs and HVs [2], [12]. Other works focus on rule-based methods that use heuristics and hand-coded rules to guide the AVs [13] or probabilistic driver modeling [14].…”

Section: Introductionmentioning

confidence: 99%

Robustness and Adaptability of Reinforcement Learning-Based Cooperative Autonomous Driving in Mixed-Autonomy Traffic

Valiente

¹

,

Toghi

²

,

Pedarsani

³

et al. 2022

IEEE Open J. Intell. Transp. Syst.

View full text Add to dashboard Cite

Building autonomous vehicles (AVs) is a complex problem, but enabling them to operate in the real world where they will be surrounded by human-driven vehicles (HVs) is extremely challenging. Prior works have shown the possibilities of creating inter-agent cooperation between a group of AVs that follow a social utility. Such altruistic AVs can form alliances and affect the behavior of HVs to achieve socially desirable outcomes. We identify two major challenges in the co-existence of AVs and HVs. First, social preferences and individual traits of a given human driver, e.g., selflessness and aggressiveness are unknown to an AV, and it is almost impossible to infer them in real-time during a short AV-HV interaction. Second, contrary to AVs that are expected to follow a policy, HVs do not necessarily follow a stationary policy and therefore are extremely hard to predict. To alleviate the above-mentioned challenges, we formulate the mixed-autonomy problem as a multi-agent reinforcement learning (MARL) problem and propose a decentralized framework and reward function for training cooperative AVs. Our approach enables AVs to learn the decision-making of HVs implicitly from experience, optimizes for a social utility while prioritizing safety and allowing adaptability; robustifying altruistic AVs to different human behaviors and constraining them to a safe action space. Finally, we investigate the robustness, safety and sensitivity of AVs to various HVs behavioral traits and present the settings in which the AVs can learn cooperative policies that are adaptable to different situations.

show abstract

“…Finally, we compared state of the art architectures related to our approach [2], [3], [18], [61] in terms of safety and efficiency to validate H3. We trained the different architectures in the same situations and examined their performance under different levels of HVs behaviors.…”

Section: ) Safetymentioning

confidence: 99%

“…In a pursuit to alleviate the challenges of this co-existence and enable social navigation for AVs, existing works either rely on models of human behavior derived from pre-recorded driving datasets [16], [17] or defining social utilities that can enforce a cooperative behavior among AVs and HVs [2], [18]. Other works focus on rule-based methods that use heuristics and hand-coded rules to guide the AVs [19] or probabilistic driver modeling [20]- [22].…”

Section: Introductionmentioning

confidence: 99%

Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic

Valiente¹,

Toghi²,

Pedarsani³

et al. 2022

Preprint

Self Cite

0

View full text Add to dashboard Cite

Building autonomous vehicles (AVs) is a complex problem, but enabling them to operate in the real world where they will be surrounded by human-driven vehicles (HVs) is extremely challenging. Prior works have shown the possibilities of creating inter-agent cooperation between a group of AVs that follow a social utility. Such altruistic AVs can form alliances and affect the behavior of HVs to achieve socially desirable outcomes. We identify two major challenges in the co-existence of AVs and HVs. First, social preferences and individual traits of a given human driver, e.g., selflessness and aggressiveness are unknown to an AV, and it is almost impossible to infer them in real-time during a short AV-HV interaction. Second, contrary to AVs that are expected to follow a policy, HVs do not necessarily follow a stationary policy and therefore are extremely hard to predict. To alleviate the above-mentioned challenges, we formulate the mixed-autonomy problem as a multi-agent reinforcement learning (MARL) problem and propose a decentralized framework and reward function for training cooperative AVs. Our approach enables AVs to learn the decision-making of HVs implicitly from experience, optimizes for a social utility while prioritizing safety and allowing adaptability; robustifying altruistic AVs to different human behaviors and constraining them to a safe action space. Finally, we investigate the robustness, safety and sensitivity of AVs to various HVs behavioral traits and present the settings in which the AVs can learn cooperative policies that are adaptable to different situations.

show abstract

“…Each player has an associated parameter, 𝛼, that indicates the player's interest in the other players achieving their objectives. This formulation relates to the Game Theoretic notion of altruism [1], as well as Social Value Orientation (SVO)-based models proposed in [12] which have garnered recent interest [14,15]. They show how this formulation can be used to efficiently compute interactive policies that account for each player's preferences.…”

Section: Introductionmentioning

confidence: 99%

Active Altruism Learning and Information Sufficiency for Autonomous Driving

Geary,

Gouk,

Ramamoorthy

2021

Preprint

0

View full text Add to dashboard Cite

Safe interaction between vehicles requires the ability to choose actions that reveal the preferences of the other vehicles. Since exploratory actions often do not directly contribute to their objective, an interactive vehicle must also able to identify when it is appropriate to perform them. In this work we demonstrate how Active Learning methods can be used to incentivise an autonomous vehicle (AV) to choose actions that reveal information about the altruistic inclinations of another vehicle. We identify a property, Information Sufficiency, that a reward function should have in order to keep exploration from unnecessarily interfering with the pursuit of an objective. We empirically demonstrate that reward functions that do not have Information Sufficiency are prone to inadequate exploration, which can result in sub-optimal behaviour. We propose a reward definition that has Information Sufficiency, and show that it facilitates an AV choosing exploratory actions to estimate altruistic tendency, whilst also compensating for the possibility of conflicting beliefs between vehicles.

show abstract

Altruistic Maneuver Planning for Cooperative Autonomous Vehicles Using Multi-agent Advantage Actor-Critic

Cited by 7 publications

References 23 publications

Robustness and Adaptability of Reinforcement Learning-Based Cooperative Autonomous Driving in Mixed-Autonomy Traffic

Robustness and Adaptability of Reinforcement Learning-Based Cooperative Autonomous Driving in Mixed-Autonomy Traffic

Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic

Active Altruism Learning and Information Sufficiency for Autonomous Driving

Contact Info

Product

Resources

About