Reinforcement Learning: A Tutorial Survey and Recent Advances

Abstract

In the last few years, Reinforcement Learning (RL), also called adaptive (or approximate) dynamic programming (ADP), has emerged as a powerful tool for solving complex sequential decision-making problems in control theory. Although seminal research in this area was performed in the artificial intelligence (AI) community, more recently, it has attracted the attention of optimization theorists because of several noteworthy success stories from operations management. It is on large-scale and complex problems of dynamic optimization, in particular the Markov decision problem (MDP) and its variants, that the power of RL becomes more obvious. It has been known for many years that on large-scale MDPs, the curse of dimensionality and the curse of modeling render classical dynamic programming (DP) ineffective. The excitement in RL stems from its direct attack on these curses, allowing it to solve problems that were considered intractable, via classical DP, in the past. The success of RL is due to its strong mathematical roots in the principles of DP, Monte Carlo simulation, function approximation, and AI. Topics treated in some detail in this survey are: Temporal differences, Q-Learning, semi-MDPs and stochastic games. Several recent advances in RL, e.g., policy gradients and hierarchical RL, are covered along with references. Pointers to numerous examples of applications are provided. This overview is aimed at uncovering the mathematical roots of this science, so that readers gain a clear understanding of the core concepts and are able to use them in their own research. The survey points to more than 100 references from the literature.

Traffic Management for Cloud Federation

“…The solution of our DP formulation searches the stochastic shortest path in a stochastic activity network [50]. This DP can be characterized as a hierarchical DP [51,52]. Therefore classical Reinforcement Learning (RL) is not suitable and hierarchical RL has to be applied [52].…”

“…Most RL approaches are based on environments that do not vary over time. We refer to [51] for a good survey on reinforcement learning techniques.…”

Real-Time QoS Control for Service Orchestration

“…The solution of our DP formulation searches the stochastic shortest path in a stochastic activity network [5]. This DP can be characterized as a hierarchical DP [8], [1]. Therefore classical Reinforcement Learning (RL) is not suitable and hierarchical RL has to be applied [1].…”

“…Most RL approaches are based on environments that do not vary over time. We refer to [8] for a good survey on reinforcement learning techniques.…”

Cooperative Reinforcement Learning Based on Zero-Sum Games

“…Instead, a learning mechanism adapting to a highly dynamic environment is more appropriate for the situation (Busoniu et al, 2008) (Xiao et al, 2007). Reinforcement learning is probably the best-known method of learning due to the similarity of empirical learning in mammals (Gosavi, 2009) (Kaelbling et al, 1996). The learning takes place by trial and error, following by performance evaluation but always provided after a sequence of actions.…”