Reinforcement Learning Agent under Partial Observability for Traffic Light Control in Presence of Gridlocks

Abstract

Bangkok is notorious for its chronic traffic congestion due to the rapid urbanization and the haphazard city plan. The Sathorn Road network area stands to be one of the most critical areas where gridlocks are a normal occurrence during rush hours. This stems from the high volume of demand imposed by the dense geographical placement of 3 big educational institutions and the insufficient link capacity with strict routes. Current solutions place heavy reliance on human traffic control expertises to prevent and disentangle gridlocks by consecutively releasing each queue length spillback through inter-junction coordination. A calibrated dataset of the Sathorn Road network area in a microscopic road traffic simulation package SUMO (Simulation of Urban MObility) is provided in the work of Chula-Sathorn SUMO Simulator (Chula-SSS). In this paper, we aim to utilize the Chula-SSS dataset with extended vehicle flows and gridlocks in order to further optimize the present traffic signal control policies with reinforcement learning approaches by an artificial agent. Reinforcement learning has been successful in a variety of domains over the past few years. While a number of researches exist on using reinforcement learning with adaptive traffic light control, existing studies often lack pragmatic considerations concerning application to the physical world especially for the traffic system infrastructure in developing countries, which suffer from constraints imposed from economic factors. The resultant limitation of the agent's partial observability of the whole network state at any specific time is imperative and cannot be overlooked. With such partial observability constraints, this paper has reported an investigation on applying the Ape-X Deep Q-Network agent at the critical junction in the morning rush hours from 6 AM to 9 AM with practically occasional presence of gridlocks. The obtainable results have shown a potential value of the agent's ability to learn despite physical limitations in the traffic light control at the considered intersection within the Sathorn gridlock area. This suggests a possibility of further investigations on agent applicability in trying to mitigate complex interconnected gridlocks in the future.

Application of Traffic Light Control in Oversaturated Urban Network Using Multi-Agent Deep Reinforcement Learning

“…The length of the collected traffic jam represents the traffic condition of every action step. Based on the study [9], we define the action step as an interval of 10 seconds. The observed state as S t = {o 1,t , .…”

Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control

“…In ( Zhang et al, 2018 ) state observability was analyzed in a vehicle-to-infrastructure (V2I) scenario, where the traffic signal agent detects approaching vehicles with Dedicated Short Range Communications (DSRC) technology under different rates. In ( Horsuwan & Aswakul, 2019 ) a scenario with partially observable state (only occupancy sensors available) was studied, however no comparisons with different state definitions or sensors were made. In ( Chu et al, 2019 ), Chu et al introduced Multiagent A2C in scenarios where different vehicle flows distributed in the network changed their insertion rates independently.…”