Learning Phase Competition for Traffic Signal Control

Abstract

Increasingly available city data and advanced learning techniques have empowered people to improve the efficiency of our city functions. Among them, improving the urban transportation efficiency is one of the most prominent topics. Recent studies have proposed to use reinforcement learning (RL) for traffic signal control. Different from traditional transportation approaches which rely heavily on prior knowledge, RL can learn directly from the feedback. On the other side, without a careful model design, existing RL methods typically take a long time to converge and the learned models may not be able to adapt to new scenarios. For example, a model that is trained well for morning traffic may not work for the afternoon traffic because the traffic flow could be reversed, resulting in a very different state representation.In this paper, we propose a novel design called FRAP, which is based on the intuitive principle of phase competition in traffic signal control: when two traffic signals conflict, priority should be given to one with larger traffic movement (i.e., higher demand). Through the phase competition modeling, our model achieves invariance to symmetrical cases such as flipping and rotation in traffic flow. By conducting comprehensive experiments, we demonstrate that our model finds better solutions than existing RL methods in the complicated all-phase selection problem, converges much faster during training, and achieves superior generalizability for different road structures and traffic conditions.

PlanLight: Learning to Optimize Traffic Signal Control With Planning and Iterative Policy Improvement

“…They also design a phase gate combined with a memory palace in the Q-network to distinguish the decision process for different phases and prevent the decision from favoring certain actions. FRAP [12] proposes a modified network structure to capture the phase competition relation between different traffic movements and speed up the training process. CoLight [15], the current stateof-the-art TSC method in multi-intersection scenario, utilizes graph attention on observations between agents to achieve cooperation.…”

“…To estimate the reward function, some methods compute the weighted sum of several measurements of traffic situations (e.g., the number of vehicles, delay, etc.) [1], [9], [20] as reward while LIT [37] proves that using the queue length as reward is equivalent to minimizing average and some works follow this idea [12], [15]. However, this statement only holds in single intersection scenario.…”

“…Recently, researchers start to investigate reinforcement learning (RL) techniques [6] for the intelligent control of traffic signals that optimize the average travel time of all passing vehicles [9], [12] in a given area. Different from supervised learning models that learn from given labeled samples, RL algorithms only require reward signals that measure the quality of single-step TSC decisions.…”

“…We record the average reward and average travel time of some typical episodes in the experiments. FRAP [12] uses negative queue length as reward and PressLight uses negative pressure as reward. As can be easily observed, the final target (average travel time) does not necessarily decrease as the acquired average reward increases.…”

Learning Traffic Signal Control from Demonstrations

“…Note that state transition matrix P is not described in model-free methods. For traffic signal control, our RL agent is defined the same as [15].…”