Learning the max pressure control for urban traffic networks considering the phase switching loss

Wang, Xingmin; Yin, Yafeng; Feng, Yiheng; Liu, Henry

doi:10.1016/j.trc.2022.103670

Cited by 29 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FRAP [82] improved RL-based methods by adding a phase competition model. PressLight [83] incorporated the method of max pressure, which aggressively selects the traffic signal phase with maximum pressure to smooth congestion [84][85][86][87][88], into the reward design and shows improvement over traditional reward design. Chen et al [89] combined the idea from FRAP [82] and PressLight [83] to achieve city-level traffic signal control.…”

Section: Literature Reviewmentioning

confidence: 99%

EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System

Su,

Zhong,

Chow

et al. 2022

Preprint

View full text Add to dashboard Cite

Emergency vehicles (EMVs) play a crucial role in responding to time-critical calls such as medical emergencies and fire outbreaks in urban areas. Existing methods for EMV dispatch typically optimize routes based on historical traffic-flow data and design traffic signal pre-emption accordingly; however, we still lack a systematic methodology to address the coupling between EMV routing and traffic signal control. In this paper, we propose EMVLight, a decentralized reinforcement learning (RL) framework for joint dynamic EMV routing and traffic signal pre-emption. We adopt the multi-agent advantage actor-critic method with policy sharing and spatial discounted factor. This framework addresses the coupling between EMV navigation and traffic signal control via an innovative design of multi-class RL agents and a novel pressure-based reward function. The proposed methodology enables EMVLight to learn network-level cooperative traffic signal phasing strategies that not only reduce EMV travel time but also shortens the travel time of non-EMVs. Simulation-based experiments indicate that EMVLight enables up to a 42.6% reduction in EMV travel time as well as an 23.5% shorter average travel time compared with existing approaches.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System

Su,

Zhong,

Chow

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Traffic simulation tools, e.g., Vissim, SUMO, CityFlow, and Aimsun, were widely used to build virtual intersections. Vehicle demand level during training was unchangeable over time , or varied by time of day [13][14][15][16][17][18][19][33][34][35][36][37][38][39][40][41]. Vehicle turning ratios were typically fixed.…”

Section: Training Environmentsmentioning

confidence: 99%

“…To choose a vehicle phase to display green with a specific duration 18 [9,[11][12][13]15,21,[26][27][28][29][31][32][33][34]36,37,39,40] To choose the green time for current vehicle phase 4 [10,17,23,41] To determine whether or not to end current vehicle phase 8 [7,16,[18][19][20]24,25,38] To adjust the green time for all vehicle phases in next cycle 5 [8,14,22,30,35] Vehicle-specific performance measure used to construct rewards Number of already served vehicles 14 [12,13,17,18,[20][21][22][23]28,31,33,34,38,39] Wait time of already ...…”

Section: Action Taken By An Agentmentioning

confidence: 99%

Double Deep Q-Network with Dynamic Bootstrapping for Real-Time Isolated Signal Control: A Traffic Engineering Perspective

Zheng

Chen

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

Real-time isolated signal control (RISC) at an intersection is of interest in the field of traffic engineering. Energizing RISC with reinforcement learning (RL) is feasible and necessary. Previous studies paid less attention to traffic engineering considerations and under-utilized traffic expertise to construct RL tasks. This study profiles the single-ring RISC problem from the perspective of traffic engineers, and improves a prevailing RL method for solving it. By qualitative applicability analysis, we choose double deep Q-network (DDQN) as the basic method. A single agent is deployed for an intersection. Reward is defined with vehicle departures to properly encourage and punish the agent’s behavior. The action is to determine the remaining green time for the current vehicle phase. State is represented in a grid-based mode. To update action values in time-varying environments, we present a temporal-difference algorithm TD(Dyn) to perform dynamic bootstrapping with the variable interval between actions selected. To accelerate training, we propose a data augmentation based on intersection symmetry. Our improved DDQN, termed D3ynQN, is subject to the signal timing constraints in engineering. The experiments at a close-to-reality intersection indicate that, by means of D3ynQN and non-delay-based reward, the agent acquires useful knowledge to significantly outperform a fully-actuated control technique in reducing average vehicle delay.

show abstract

“…The methods (Le et al., 2015; Levin et al., 2020) applied “cyclic phases” policy that arranges the phases into a fixed‐ordered sequence. In recent studies, the back‐pressure approach has been combined with model‐based optimal control methods or RL methods (D. Ma et al., 2020; Wang et al., 2022). A summary of significant publications on max‐pressure methods is presented in Table 1.…”

Section: Introductionmentioning

confidence: 99%

Field‐tested signal controller to mitigate spillover using trajectory data

Han,

Ding

et al. 2023

Computer aided Civil Eng

Self Cite

View full text Add to dashboard Cite

Trajectory data from connected vehicles (CVs) provide a continuous and reliable means of obtaining information that can be leveraged to optimize traffic signals. This paper proposes a real‐time traffic signal control method using CV trajectory data as the sole input. The primary goal of the proposed signal control method is to prevent queue spillover, which may significantly decrease the traffic efficiency on urban networks and induce high delays to the travelers. The proposed method formulates the signal control problem via a linear quadratic optimization model, considering the constraints related to the duration and variability of green lights in practical traffic signal control systems. Compared to conventional max‐pressure‐based methods, the optimization model offers enhanced efficiency in handling these constraints, making it highly suitable for real‐life implementation. The proposed method has undergone testing in both simulated environments and real‐world applications. In the simulation experiments, the proposed method has been demonstrated to effectively reduce spillover risks and outperform a conventional max‐pressure‐based approach even when the CV penetration rate is as low as 5%. In the real‐world experiment, the proposed method had been tested in a traffic network around the Beijing Capital International Airport for several months. The severity of spillovers, which was represented by two performance indicators, had been significantly reduced after implementing the proposed method.

show abstract

Learning the max pressure control for urban traffic networks considering the phase switching loss

Cited by 29 publications

References 20 publications

EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System

EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System

Double Deep Q-Network with Dynamic Bootstrapping for Real-Time Isolated Signal Control: A Traffic Engineering Perspective

Field‐tested signal controller to mitigate spillover using trajectory data

Contact Info

Product

Resources

About