Deep Reinforcement Meta-Learning and Self-Organization in Complex Systems: Applications to Traffic Signal Control

Abstract

We studied the ability of deep reinforcement learning and self-organizing approaches to adapt to dynamic complex systems, using the applied example of traffic signal control in a simulated urban environment. We highlight the general limitations of deep learning for control in complex systems, even when employing state-of-the-art meta-learning methods, and contrast it with self-organization-based methods. Accordingly, we argue that complex systems are a good and challenging study environment for developing and improving meta-learning approaches. At the same time, we point to the importance of baselines to which meta-learning methods can be compared and present a self-organizing analytic traffic signal control that outperforms state-of-the-art meta-learning in some scenarios. We also show that meta-learning methods outperform classical learning methods in our simulated environment (around 1.5–2× improvement, in most scenarios). Our conclusions are that, in order to develop effective meta-learning methods that are able to adapt to a variety of conditions, it is necessary to test them in demanding, complex settings (such as, for example, urban traffic control) and compare them against established methods.

Democratizing Traffic Control in Smart Cities

“…To illustrate this, we will consider traffic flows in urban street networks, which were initially uncontrolled (or controlled by static traffic signs) (Mueller, 1970), then controlled by human traffic signal operators (Hamilton et al, 2013), later replaced by cyclical signaling schemes (Urbanik et al, 2015), then replaced by algorithmic systems (Lämmer and Helbing, 2008), which are now increasingly replaced by AI methods (often based on Deep Reinforcement Learning) (Korecki, 2023).…”