Twin-Delayed Deep Deterministic Policy Gradient for Low-Frequency Oscillation Damping Control

Abstract

Due to the large scale of power systems, latency uncertainty in communications can cause severe problems in wide-area measurement systems. To resolve this issue, a significant amount of past work focuses on using emerging technology, including machine learning methods such as Q-learning, for addressing latency issues in modern controls. Although the method can deal with the stochastic characteristics of communication latency, the Q-values can be overestimated in Q-learning methods, leading to high bias. To address the overestimation bias issue, we redesign the learning structure of the deep deterministic policy gradient (DDPG). Then we develop a damping control twin-delayed deep deterministic policy gradient method to handle the damping control issue under unknown latency in the power network. The purpose is to address the damping control issue under unknown latency in the power network. This paper will create a novel reward algorithm, taking into account the machine speed deviation, the episode termination prevention, and the feedback from action space. In this way, the system optimally damps down frequency oscillations while maintaining the system’s stability and reliable operation within defined limits. The simulation results verify the proposed algorithm in various perspectives, including the latency sensitivity analysis under high renewable energy penetration and the comparison with conventional and machine learning control algorithms. The proposed method shows a fast learning curve and good control performance under varying communication latency.

Power Allocation Strategy for Urban Rail HESS Based on Deep Reinforcement Learning Sequential Decision Optimization

“…Currently, the twin-delayed deep deterministic policy gradient (TD3) is the more popular algorithm in DRL [27] [28], which has been successfully applied in many different areas [29][30][31][32][33][34]. Given the time-varying characteristics in urban rail transit systems, we propose a modified TD3 algorithm by introducing a priority experience sampling strategy, termed A-TD3 (Annealing bias-priority experience replay twin delayed deep deterministic policy gradient algorithm).…”

Optimal Inter-area Oscillation Damping Control: A Transfer Deep Reinforcement Learning Approach with Switching Control Strategy

“…Meanwhile, DRL is robust to the uncertainties of the system, i.e., control signals latency, environment noise, and load variations. A Deep Deterministic Policy Gradient (DDPG) based method and the twin-delayed DDPG method are proposed to overcome various communication delays during damping control [11], [12]. To solve the high dimensionality problem of power systems, Mukherjee et al [13] introduce two model reduction approaches for scalable DRL wide-area damping control.…”

Two-Loop Acceleration Autopilot Design and Analysis Based on TD3 Strategy

“…Motivated by the previous investigations, this paper employs the DRL principle to design a parameter tuning model for a two-loop autopilot using the TD3 algorithm [33], which has the following striking advantages over existing autopilot design schemes:…”