Multi-Intersection Control with Deep Reinforcement Learning and Ring-and-Barrier Controllers

Abstract

This paper discusses a machine-learning traffic signal control method. A full-scale corridor is analyzed and the transferability of using a model pre-trained on a single intersection is examined. Two controller designs are explored, a simple two-phase design and a full ring-and-barrier style controller. The full ring-and-barrier controller adapts many of the key features present in traditional controllers, such as protected-permissive left turns, so that they can be used in the reinforcement learning (RL) paradigm. This study is the first to propose a method that uses deep reinforcement learning (DRL) to implement a full ring-and-barrier style controller. The study also examines the feasibility of using transfer learning to pre-train a model on a single intersection and then fine-tune it for application in a complete environment. Training is done on a simple four lane intersection and the pre-trained model is then transferred for fine-tuning to six controllers operating on a corridor modeled with field data obtained for University Avenue in Waterloo, Ontario, Canada. The performance of the fully trained model is then compared with the existing signal plans in relation to the average delay and average queue length. Application of the ring-and-barrier design to this corridor was found to reduce delays by at least 5% and average queue lengths at intersections by 27%.

Effects analysis of reward functions on reinforcement learning for traffic signal control

“…To reflect a separate left-turn, few studies have implemented the ring-and-barrier system. Muresan et al [ 25 ] proposed deep RL methods to implement simple two-phase and full ring-and-barrier controllers. The application of the ring-and-barrier design reduced delays by at least 5% and average queue lengths at intersections by 27%.…”

“…The action definitions of RL-TSC include phase switching, phase selection, and phase duration. Phase switching involves expanding the current phase or skipping unnecessary phases in response to changes in traffic conditions [ 5 , 6 , 10 , 13 , 24 , 25 , 28 ]. This method cannot change the order of phase sequences unless the phase is skipped.…”

“…Our models adopt the deep deterministic policy gradient algorithm, which is one of the policy gradient methods of deep RL. It can use a continuous action space unlike the deep Q-network algorithm [ 8 , 13 , 24 , 25 ]. It is suitable for calculating traffic signals that needs to respond to dynamic traffic flow changes.…”

Extensible prototype learning for real‐time traffic signal control

“…Muresan et al. (2021) proposed multi‐intersection control with a fine‐tuning technique to apply a pre‐trained model to different intersections. Ge et al.…”

“…Muresan et al. (2021) were the first to propose a model using deep RL in a full ring‐and‐barrier style controller. They defined the “optional” and “mandatory” phases based on the minimum green time conditions such as the pedestrian green interval.…”