Reinforcement Learning and Dynamic Programming Using Function Approximators

Abstract

Control systems are making a tremendous impact on our society. Though invisible to most users, they are essential for the operation of nearly all devices -from basic home appliances to aircraft and nuclear power plants. Apart from technical systems, the principles of control are routinely applied and exploited in a variety of disciplines such as economics, medicine, social sciences, and artificial intelligence.A common denominator in the diverse applications of control is the need to influence or modify the behavior of dynamic systems to attain prespecified goals. One approach to achieve this is to assign a numerical performance index to each state trajectory of the system. The control problem is then solved by searching for a control policy that drives the system along trajectories corresponding to the best value of the performance index. This approach essentially reduces the problem of finding good control policies to the search for solutions of a mathematical optimization problem.Early work in the field of optimal control dates back to the 1940s with the pioneering research of Pontryagin and Bellman. Dynamic programming (DP), introduced by Bellman, is still among the state-of-the-art tools commonly used to solve optimal control problems when a system model is available. The alternative idea of finding a solution in the absence of a model was explored as early as the 1960s. In the 1980s, a revival of interest in this model-free paradigm led to the development of the field of reinforcement learning (RL). The central theme in RL research is the design of algorithms that learn control policies solely from the knowledge of transition samples or trajectories, which are collected beforehand or by online interaction with the system. Most approaches developed to tackle the RL problem are closely related to DP algorithms.A core obstacle in DP and RL is that solutions cannot be represented exactly for problems with large discrete state-action spaces or continuous spaces. Instead, compact representations relying on function approximators must be used. This challenge was already recognized while the first DP techniques were being developed. However, it has only been in recent years -and largely in correlation with the advance of RL -that approximation-based methods have grown in diversity, maturity, and efficiency, enabling RL and DP to scale up to realistic problems.This book provides an accessible in-depth treatment of reinforcement learning and dynamic programming methods using function approximators. We start with a concise introduction to classical DP and RL, in order to build the foundation for the remainder of the book. Next, we present an extensive review of state-of-the-art approaches to DP and RL with approximation. Theoretical guarantees are provided on the solutions obtained, and numerical examples and comparisons are used to illustrate the properties of the individual methods. The remaining three chapters are i ii dedicated to a detailed presentation of representative algorithms from the three major classes o...

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

“…Encouraged by these successes, the Deep RL field of research is revisiting some of the main works achieved in the past three decades using deep learning function approximators (see Busoniu et al (2010) for a global view of the use of function approximators in RL). In particular, the deep learning remastering of Double Q Learning (see Hasselt et al (2015) and Hasselt (2010) for the original version without deep learning) or Memory Replay Schaul et al (2015) ideas shows that RL promises that were originally proposed decades ago are definitely worth revisiting in the light of Deep Learning architecture.…”

“…After each moment of interaction, the agent receives a feedback signal, reinforcement signal, or reward from the system being delivered to the learning system in response to the execution of control action Sutton and Barto, 1998;Busoniu et al, 2010. The most commonly studied objective is to maximize, for each time step, the expected sum of future reinforcements or discounted return defined as the sum of rewards over future time steps Sutton and Barto, 1998;Busoniu et al, 2010. A likely framework for application of the RL in power system decision and control is illustrated in Fig. 3.…”

“…They process the accumulated experience and progressively learn an appropriate control law, i.e. an algorithm to associate suitable actions to the observations in order to fulfil a pre-specified objective Sutton and Barto (1998); Busoniu et al (2010).…”

“…RL offers a panel of methods that allows controllers to learn a goal-oriented control law from interactions with a system or its simulation model Sutton and Barto (1998); Busoniu et al (2010). RL driven controllers (agents) observe the system state, take actions, and observe the effects of these actions.…”

An Intelligent Strategy Based on Fuzzy Reinforcement Learning for Backing Truck up

“…It can also be simply combined with fuzzy logic and provide the relationship between the states and the accessible action, which is the same as creating the fuzzy logic "if. ..then" engine [23][24][25].…”