Application of reinforcement learning with continuous state space to ramp metering in real-world conditions

Abstract

Abstract² In this paper we introduce a new approach to Freeway Ramp Metering (RM) based on Reinforcement Learning (RL) with focus on real-life experiments in a case study in the City of Toronto. Typical RL methods consider discrete state representation that lead to slow convergence in complex problems. Continuous representation of state space has the potential to significantly improve the learning speed and therefore enables tackling large-scale complex problems. A robust approach based on local regression, named k nearest neighbors temporal difference (kNN-TD), is employed to represent state space continuously in the RL environment. The performance of the new algorithm is compared against the ALINEA controller and typical RL methods using a microsimulation testbed in Paramics. The results show that RM using the kNN-TD method can reduce total network travel time by 44% compared to the do-nothing case (without RM) and by 17% compared to ALINEA.

Automated Vehicle Control at Freeway Lane-drops: a Deep Reinforcement Learning Approach

“…Jacob and Abdulhai (2010) presented a methodology that used RL to control freeway-corridor traffic, by integrating ramp metering and route guidance via variable message signs (VMS). Davarynejad et al 2011, and Rezaee et al (2012), applied RL to ramp metering. El-Tantawy et al (2013) developed a control strategy that integrated a network of adaptive traffic signal controllers using a multi-agent RL, while Fares and Gomaa (2014) used RL approach to control traffic at an isolated on-ramp.…”

Reinforcement Learning for Ramp Control: An Analysis of Learning Parameters

“…If the algorithm selects higher ε values such as 0.5 and 0.9, it will fail to reach the benchmark line. [7][8][9][10][11][12][13], α is often less than 0.5, γ is between 0.7 and 0.9 and ε is usually around 0.1.…”

“…After this contribution, some recent studies have also shown the effectiveness of RL for ramp control under different settings and conditions. For instance, coordinated ramp control using RL is considered in [9], continuous state space was analyzed in [10], and indirect RL was tested in [11,12]. Although some efforts have been made to explore the application of RL in the ramp control domain, the issues of how to set the parameters for RL based ramp control strategies and how these settings influence the algorithm performance have not been widely studied.…”

Deep reinforcement learning based conflict detection and resolution in air traffic control

“…Reinforcement learning (RL) has been utilised in many transportation fields. For example, it has been used in urban traffic light control [13–15] and ramp metering in highway [16–18]. In air traffic management field, a series of achievements have been made with RL.…”