Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem

Ernst, Damien; Glavić, Mevludin; Capitanescu, Florin; Wehenkel, Louis

doi:10.1109/tsmcb.2008.2007630

Cited by 183 publications

(113 citation statements)

References 45 publications

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…The work presented in Glavic et al (2005a) suggested fusion of RL and the concept of control Lyapunov functions (this approach is further elaborated in Glavic et al (2006)). Other works suggesting fusion of RL with known control techniques include Ernst et al (2009);Wang et al (2014) where RL was considered together with model predictive control, and Li and Wu (1999) where RL is combined with fuzzy logic control.…”

Section: Past and Recent Considerations Of Rl For Electric Power Systmentioning

confidence: 99%

“…Power system components considered include: dynamic brake Ernst et al (2004); Glavic (2005), thyristor controlled series capacitor Ernst et al (2004Ernst et al ( , 2009, quadrature booster Li and Wu (1999), synchronous generators (all AGC related references), individual or aggregated loads Vandael et al (2015); Ruelens et al (2016), etc. If used as a multi-agent system, then additional state variables must be introduced to ensure convergence of these essentially distributed computation schemes, and an adapted variant of standard RL methods is often used (for example correlated equilibrium Q(λ) Yu et al (2012a)).…”

Section: Past and Recent Considerations Of Rl For Electric Power Systmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

2017

View full text Add to dashboard Cite

Abstract:In this paper, we review past (including very recent) research considerations in using reinforcement learning (RL) to solve electric power system decision and control problems. The RL considerations are reviewed in terms of specific electric power system problems, type of control and RL method used. We also provide observations about past considerations based on a comprehensive review of available publications. The review reveals the RL is considered as viable solutions to many decision and control problems across different time scales and electric power system states. Furthermore, we analyse the perspectives of RL approaches in light of the emergence of new-generation, communications, and instrumentation technologies currently in use, or available for future use, in power systems. The perspectives are also analysed in terms of recent breakthroughs in RL algorithms (Safe RL, Deep RL and path integral control for RL) and other, not previously considered, problems for RL considerations (most notably restorative, emergency controls together with so-called system integrity protection schemes, fusion with existing robust controls, and combining preventive and emergency control).

show abstract

Section: Past and Recent Considerations Of Rl For Electric Power Systmentioning

confidence: 99%

Section: Past and Recent Considerations Of Rl For Electric Power Systmentioning

confidence: 99%

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

2017

View full text Add to dashboard Cite

show abstract

“…Model Predictive Control techniques have originally been introduced as ways to stabilize large-scale systems with constraints around equilibrium points (or around a reference trajectory) [36,10,15]. They exploit an explicitly formulated model of the problem and solve in a receding horizon manner a series of finite time open-loop deterministic optimal control problems.…”

Section: Model Predictive Controlmentioning

confidence: 99%

Optimized look‐ahead tree policies: a bridge between look‐ahead tree policies and direct policy search

Jung

Wehenkel

Ernst

et al. 2013

Adaptive Control & Signal

View full text Add to dashboard Cite

Direct policy search (DPS) and look-ahead tree (LT) policies are two widely used classes of techniques to produce high performance policies for sequential decision-making problems. To make DPS approaches work well, one crucial issue is to select an appropriate space of parameterized policies with respect to the targeted problem. A fundamental issue in LT approaches is that, to take good decisions, such policies must develop very large look-ahead trees which may require excessive online computational resources. In this paper, we propose a new hybrid policy learning scheme that lies at the intersection of DPS and LT, in which the policy is an algorithm that develops a small look-ahead tree in a directed way, guided by a node scoring function that is learned through DPS. The LT-based representation is shown to be a versatile way of representing policies in a DPS scheme, while at the same time, DPS enables to significantly reduce the size of the look-ahead trees that are required to take high-quality decisions.We experimentally compare our method with two other state-of-the-art DPS techniques and four common LT policies on four benchmark domains and show that it combines the advantages of the two techniques from which it originates. In particular, we show that our method: (1) produces overall better performing policies than both pure DPS and pure LT policies, (2) requires a substantially smaller number of policy evaluations than other DPS techniques, (3) is easy to tune and (4) results in policies that are quite robust with respect to perturbations of the initial conditions.

show abstract

“…However, these strategies usually consider problems with finite state spaces where the uncertainities come from the lack of knowledge of the transition probabilities [7,5]. In model predictive control (MPC) where the environment is supposed to be fully known [10], min max approaches have been used to determine the optimal sequence of actions with respect to the "worst case" disturbance sequence occuring [1]. The CGRL algorithm relies on a methodology for computing a lower bound on the worst possible return (considering any compatible environment) in a deterministic setting with a mostly unknown actual environment.…”

Section: Related Workmentioning

confidence: 99%

Towards Min Max Generalization in Reinforcement Learning

Fonteneau

Murphy

Wehenkel

et al. 2011

Communications in Computer and Information Science

View full text Add to dashboard Cite

Abstract. In this paper, we introduce a min max approach for addressing the generalization problem in Reinforcement Learning. The min max approach works by determining a sequence of actions that maximizes the worst return that could possibly be obtained considering any dynamics and reward function compatible with the sample of trajectories and some prior knowledge on the environment. We consider the particular case of deterministic Lipschitz continuous environments over continuous state spaces, finite action spaces, and a finite optimization horizon. We discuss the non-triviality of computing an exact solution of the min max problem even after reformulating it so as to avoid search in function spaces. For addressing this problem, we propose to replace, inside this min max problem, the search for the worst environment given a sequence of actions by an expression that lower bounds the worst return that can be obtained for a given sequence of actions. This lower bound has a tightness that depends on the sample sparsity. From there, we propose an algorithm of polynomial complexity that returns a sequence of actions leading to the maximization of this lower bound. We give a condition on the sample sparsity ensuring that, for a given initial state, the proposed algorithm produces an optimal sequence of actions in open-loop. Our experiments show that this algorithm can lead to more cautious policies than algorithms combining dynamic programming with function approximators.

show abstract

Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem

Cited by 183 publications

References 45 publications

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

Optimized look‐ahead tree policies: a bridge between look‐ahead tree policies and direct policy search

Towards Min Max Generalization in Reinforcement Learning

Contact Info

Product

Resources

About