Application of Deep Reinforcement Learning in Traffic Signal Control: An Overview and Impact of Open Traffic Data

Abstract

Persistent congestions which are varying in strength and duration in the dense traffic networks are the most prominent obstacle towards sustainable mobility. Those types of congestions cannot be adequately resolved by the traditional Adaptive Traffic Signal Control (ATSC). The introduction of Reinforcement Learning (RL) in ATSC as tackled those types of congestions by using on-line learning, which is based on the trial and error approach. Furthermore, RL is prone to the dimensionality curse related to the state–action space size based on which a non-linear quality function is derived. The Deep Reinforcement Learning (DRL) framework uses Deep Neural Networks (DNN) to digest raw traffic data to approximate the quality function of RL. This paper provides a comprehensive analysis of the most recent DRL approaches used for the ATSC algorithm design. Special emphasis is set to overview of the traffic state representation and multi-agent DRL frameworks applied for the large traffic networks. Best practices are provided for choosing the adequate DRL model, hyper-parameters tuning, and model architecture design. Finally, this paper provides a discussion about the importance of the open traffic data concept for the extensive application of DRL in the real world ATSC.

Reducing Congestion in an Intelligent Traffic System With Collaborative and Adaptive Signaling on the Edge

“…Model-Free Reinforcement Learning is an answer to optimization problems of those Markov Decision Processes for which state space is not fully observable, or a mathematical model is not well-developed or well-understood. A good sur- (Table 2) vey on the application of RL methods to the adaptive traffic control systems is presented in [18,26]. Recently, Li et al [34] claimed that DNNs could be used to learn the dynamics of a traffic system and defined a signal plan by modeling the control action and system state.…”

“…However, using the average wait time for RL Reward has its merits, which is evaluated later in this article. A method to estimate FIGURE 4: RL Architecture with reference to Deep Q-Network Algorithm [26] the same is also presented. CASE strives to maximize the reciprocal of this wait time as a reward.…”

“…• Scale of Horizon: Ideally, a truly optimal SPaT pattern should consider all the intersections of a road network. Alternatively, considering only local state-space at a road intersection is faster to process but may not converge well [26]. A balanced horizon is required that includes a subset of nodes that effectively reflect the system state.…”

Self-Adaptive Priority Correction for Prioritized Experience Replay

“…As a pivotal field of machine learning, deep reinforcement learning (DRL) has surpassed the human level in many fields, such as video games [1], robot control [2,3], traffic signal control [4], chess tasks [5], and speech recognition [6]. The typical process for DRL is as follows.…”

MTGPLP Approach for Traffic Signal Intelligent Control

“…For optimizing the traffic signal at a single intersection, neural networks methods and convolution neural networks approaches were utilized to optimize the traffic signal plans [44][45][46][47]. In addition, deep reinforcement learning techniques were applied to solve the traffic signal schemes [48][49][50][51]. Furthermore, the optimum traffic signal timing plans were governed with Q-Learning [52][53][54][55][56].…”