Trajectory Optimization of CAVs in Freeway Work Zone considering Car-Following Behaviors Using Online Multiagent Reinforcement Learning

Abstract

Work zone areas are frequent congested sections considered as the freeway bottleneck. Connected and autonomous vehicle (CAV) trajectory optimization can improve the operating efficiency in bottleneck areas by harmonizing vehicles’ manipulations. This study presents a joint trajectory optimization of cooperative lane changing, merging, and car-following actions for CAV control at a local merging point together with upstream points. The multiagent reinforcement learning (MARL) method is applied in this system, with one agent providing a merging advisory service at the merging point and controlling the inner-lane vehicles’ headway for smooth outer-lane vehicle merging, while other agents provide lane-changing advisory services at advance lane-changing points to control how vehicles make lane changes in advance and perform corresponding headway adjustment, similar to and jointly with the merging advisory service. Uniting all agents, the coordination graph (CG) method is applied to seek the global optimum, overcoming the exponential growth problem in MARL. Using MATLAB and the VISSIM COM interface, an online simulation platform is established. The simulation results show that MARL is effective for online computation with in-timing response. More importantly, comparisons of the results obtained in various scenarios demonstrate that the proposed system obtained smoother vehicle trajectories in all controlled sections, rather than only in the merging area, indicating that it can achieve better traffic conditions in freeway work zone areas.

Review of driving-behaviour simulation: VISSIM and artificial intelligence approach

“… The average delay and average queue length decreased when the algorithm was modified. [ 189 ] Visualisations of real-time trajectory optimisation for each vehicle. Multiagent Reinforcement Learning Algorithm (MARL).…”

“…used VISSIM as an agent-learning environment by employing the deep Q-learning algorithm to control a signal at an isolated intersection. The simulation environment allowed the agent to learn before being subjected to a real-world test [ 189 ]. used the final approach for calculation and visualisation to regulate the fluctuating headway in a highway work zone region by utilising VISSIM COM interface and multi-agent reinforcement learning algorithm and a simulation platform to visualise real-time trajectory for each vehicle optimisation.…”

A Novel Dynamic Lane-Changing Trajectory Planning Model for Automated Vehicles Based on Reinforcement Learning

“…In recent years, RL is increasingly used in vehicle control problems in ramp metering [26,27], intersection [28,29], and freeway work zone [30], in order to improve traffic conditions. RL-based control models for AVs have a positive impact on traffic safety and efficiency [25,31,32].…”

Using Reinforcement Learning to Handle the Unintended Lateral Attack in the Intelligent Connected Vehicle Environment

“…In recent years, reinforcement learning has been applied in the game of Go [11], highly automated driving [12][13][14], trafc signal control [15][16][17], and has proven its efectiveness. Meanwhile, multi-agent reinforcement learning is becoming a hot research area [18][19][20]. Whatever single-agent or multiagent reinforcement learning, there are some basic and commonly used reinforcement learning methods like DQN (Deep Q Networks), PG (Policy Gradient), DDPG (Deep Deterministic Policy Gradient), TD3 (Twin Delayed DDPG), SAC (Soft Actor Critic), and A2C (Advantage Actor Critic).…”