Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

Chen, Dong; Li, Zhaojian; Wang, Yongqiang; Jiang, Longsheng; Wang, Yue

doi:10.48550/arxiv.2105.05701

Cited by 18 publications

(33 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, multi-agent reinforcement learning (MARL) algorithms have also been explored for autonomous driving tasks [13,14,22,23]. An MARL algorithm with hard-coded safety constraints [13] was proposed to solve the double-merge problem.…”

Section: Data-driven Methodsmentioning

confidence: 99%

“…In such a framework, a hierarchical temporal abstraction method was applied to reduce the effective horizon and the variance of the gradient estimation error. In the recent literature [23], an MARL algorithm was delivered to solve the onramp merging problem with safety enhancement by a novel priority-based safety supervisor. In addition, a novel MARL approach [22] was realized with the combination of Graphic Convolution Neural Network (GCN) [24] and Deep Q Network (DQN) [25] to better fuse the acquired information from collaborative sensing, showing promising results on a 3-lane freeway containing 2 off-ramps highway environment.…”

Section: Data-driven Methodsmentioning

confidence: 99%

“…The goal in cooperative MARL setting is to maximize the global reward of all the agents. To overcome the communication overhead and the credit assignment problem [34], we adopt the local reward design [23] as follows:…”

Section: Ma2c For Avsmentioning

confidence: 99%

“…Then the shared actor-critic network will update the policy and value networks with the extracted features. As mentioned in the recent literature [23], the adopted parameter sharing scheme [12] between the actor and value networks can greatly improve the learning efficiency.…”

Section: Ma2c For Avsmentioning

confidence: 99%

“…The reason is that, in separate actor-critic networks, the critic network can only guide the actor network to the correct training direction until the critic network is well-trained which may take a long time to achieve. In contrast, the actor network can benefit from the shared state representation via the critic network in a shared actor-critic network [23,39].…”

Section: Sharing Vs Separate Actor-critic Networkmentioning

confidence: 99%

See 4 more Smart Citations

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Zhou

Chen

Yan

et al. 2022

Auton. Intell. Syst.

Self Cite

View full text Add to dashboard Cite

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL) has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision-making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic (MA2C) method is proposed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is designed to incorporate fuel efficiency, driving comfort, and the safety of autonomous driving. A comprehensive experimental study is made that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety, and driver comfort.

show abstract

Section: Data-driven Methodsmentioning

confidence: 99%

Section: Data-driven Methodsmentioning

confidence: 99%

Section: Ma2c For Avsmentioning

confidence: 99%

Section: Ma2c For Avsmentioning

confidence: 99%

Section: Sharing Vs Separate Actor-critic Networkmentioning

confidence: 99%

See 3 more Smart Citations

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Zhou

Chen

Yan

et al. 2022

Auton. Intell. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

TMPF: A Two‐Stage Merging Planning Framework for Dense Traffic

Chen

Cao

Guo

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Planning for autonomous vehicles to merge into high‐density traffic flows within limited mileage is quite challenging. Specifically, the driving trajectory will inevitably have intersections with other vehicles whose driving intentions can't be directly observed. Herein, a two‐stage algorithm framework that is decomposed into the longitudinal and lateral planning processes for online merging planning is proposed. An improved particle filter is used to estimate the driving models of surrounding vehicles for predicting their future driving intentions. Based on Monte Carlo tree search (MCTS), different action spaces are evaluated for longitudinal merging gap selection and lateral interactive merging operation, while heuristic pruning is used to reduce the computation cost. Moreover, the coefficients related to the driving styles are introduced, and their influences on merging performance are analyzed. Finally, the proposed algorithm is implemented in a two‐lane simulation environment. The results show that the proposal has outperformed other baseline methods.

show abstract

Safe Lane-Changing in CAVs Using External Safety Supervisors: A Review

Lenka

Bouroche

2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Connected autonomous vehicles (CAVs) can exploit information received from other vehicles in addition to their sensor information to make decisions. For this reason, their deployment is expected to improve traffic safety and efficiency. Safe lane-changing is a significant challenge for CAVs, particularly in mixed traffic, i.e. with human-driven vehicles (HDVs) on the road, as the set of vehicles around them varies very quickly, and they can only communicate with a fraction of them. Many approaches have been proposed, with most recent work adopting a multi-agent reinforcement learning (MARL) approach, but those do not provide safety guarantees making them unsuitable for such a safety-critical application. A number of external safety techniques for reinforcement learning have been proposed, such as shielding, control barrier functions, model predictive control and recovery RL, but those have not been applied to CAV lane changing.This paper investigates whether external safety supervisors could be used to provide safety guarantees for MARL-based CAV lane changing (LC-CAV). For this purpose, a MARL approach to CAV lane changing (MARL-CAV) is designed, using parameter sharing and a replay buffer to motivate cooperative behaviour and collaboration among CAVs. This is then used as a baseline to discuss the applicability of the state-of-the-art external safety techniques for reinforcement learning to MARL-CAV. Comprehensive analysis shows that integrating an external safety technique to MARL for lane changing in CAVs is challenging, and none of the existing external safety techniques can be directly applied to MARL-CAV as these safety techniques require prior knowledge of unsafe states and recovery policies.

show abstract

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

Cited by 18 publications

References 41 publications

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

TMPF: A Two‐Stage Merging Planning Framework for Dense Traffic

Safe Lane-Changing in CAVs Using External Safety Supervisors: A Review

Contact Info

Product

Resources

About