Reinforcement Learning for Optimal Primary Frequency Control: A Lyapunov Approach

Abstract

The increase in penetration of inverter-based resources provide us with more flexibility in frequency regulation of power systems in addition to conventional linear droop controllers. Because of the fast power electronic interfaces, inverterbased resources can be used to realize complex control functions and potentially offer large gains in performance compared to linear controllers. Reinforcement learning has emerged as popular method to find these nonlinear controllers by parameterizing them as neural networks.The key challenge with learning based approach is that stability constraints are difficult to enforce on the learned controllers. In addition, optimizing the controllers are nontrivial because of the time-coupled dynamics of power systems. In this paper, we propose to explicitly engineer the structure of neural network based controllers such that they guarantee system stability for all topologies and parameters. This is done by using a Lyapunov function to guide their structures. A recurrent neural network based reinforcement learning architecture is used to efficiently train the weights of controllers. The resulting controllers only use local information and outperform linear droop as well as strategies learned purely by using reinforcement learning.

Stability Constrained Reinforcement Learning for Real-Time Voltage Control

“…However, their approach only applies to the frequency control application, while our method works for voltage control which requires a different Lyapunov function design. Interestingly, both our work and prior work [37] arrive at a similar stability condition, that is strict policy monotonicity guarantees system stability.…”

“…There are different approaches for monotone neural network architecture design in literature [37], [41], [42]. In this paper we follow the monotonic neural network design in [37, Lemma 3], which used a single hidden layer neural network with d hidden units and ReLU activation.…”

“…In particular, our experiments compare against [21], which uses standard multi-agent DDPG for voltage control. Closest in spirit to our paper is [37], which proposes a stable RL approach for frequency control via a Lyapunov approach. However, their approach only applies to the frequency control application, while our method works for voltage control which requires a different Lyapunov function design.…”

Decentralized Safe Reinforcement Learning for Voltage Control

“…To this end, we formulate the controller with a stacked-ReLU structure shown in Fig. 3, which is developed in [27]. This design is a piecewise linear function where the slope of each piece is equal to the summation of weights in activated neurons.…”

Understanding the Safety Requirements for Learning-based Power Systems Operations

“…In addition, compared to rule-based algorithms, RL methods can learn directly from experiences without constructing an expert dataset [2], [10]. Learning-based control paradigms have been proposed for a variety of operation tasks in power grids, including control of voltage and frequency [11]- [14], capacity scheduling of PV and energy storage [15], topology control [1] and many more. A more detailed review of RL for power system operation can be found in [10].…”