Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning

Abstract

Lane-change maneuvers are commonly executed by drivers to follow a certain routing plan, overtake a slower vehicle, adapt to a merging lane ahead, etc. However, improper lane change behaviors can be a major cause of traffic flow disruptions and even crashes. While many rule-based methods have been proposed to solve lane change problems for autonomous driving, they tend to exhibit limited performance due to the uncertainty and complexity of the driving environment. Machine learning-based methods offer an alternative approach, as Deep reinforcement learning (DRL) has shown promising success in many application domains including robotic manipulation, navigation, and playing video games. However, applying DRL for autonomous driving still faces many practical challenges in terms of slow learning rates, sample inefficiency, and non-stationary trajectories. In this study, we propose an automated lane change strategy using proximal policy optimization-based deep reinforcement learning, which shows great advantage in learning efficiency while maintaining stable performance. The trained agent is able to learn a smooth, safe, and efficient driving policy to determine lane-change decisions (i.e. when and how) even in dense traffic scenarios. The effectiveness of the proposed policy is validated using task success rate and collision rate, which demonstrates the lane change maneuvers can be efficiently learned and executed in a safe, smooth and efficient manner.

An enhanced eco-driving strategy based on reinforcement learning for connected electric vehicles: cooperative velocity and lane-changing control

“…Comfort: jerk , the change rate of acceleration, is regarded as a general indicator to evaluate comfort (Guo et al , 2021; Ye et al , 2020). …”

“…To make the energy consumption value as small as possible, we set the sub-reward function to the reciprocal of the above formula: Travel efficiency: velocity, a simple and effective indicator, can be neatly used to represent the transport efficiency (He et al , 2020; Jan et al , 2020). Comfort: jerk , the change rate of acceleration, is regarded as a general indicator to evaluate comfort (Guo et al , 2021; Ye et al , 2020). …”

“…Krasowski et al (2020) extended RL with a safety layer which limits the action space into the sub-action of safe actions to address the safety problem of autonomous vehicles. Ye et al (2020) used proximal policy optimization (PPO) to realize automatic lane-changing strategy. The results show that this method can learn and execute lane-changing actions safely, stably and efficiently.…”

Proposing Cooperative Maneuvers Among Automated Vehicles Using Machine Learning

“…In an early study, Moriarty and Langley [15] use reinforcement learning (RL) to derive lane changing decisions optimizing highway traffic flow. Ye et al [18] let vehicles learn smooth and efficient, collision-free lane change trajectories via RL. Chandramohan et al [5] use Deep Q-Learning to enable an automated vehicle to avoid collisions on a multi-lane highway.…”

“…Finding driving strategies often involves machine learning (ML) [5,15,17,18]. We investigate whether ML can also help make decisions on cooperative maneuvers.…”