Voltage Control-Based Ancillary Service Using Deep Reinforcement Learning

Abstract

Ancillary services rely on operating reserves to support an uninterrupted electricity supply that meets demand. One of the hidden reserves of the grid is in thermostatically controlled loads. To efficiently exploit these reserves, a new realization of control of voltage in the allowable range to follow the set power reference is proposed. The proposed approach is based on the deep reinforcement learning (RL) algorithm. Double DQN is utilized because of the proven state-of-the-art level of performance in complex control tasks, native handling of continuous environment state variables, and model-free application of the trained DDQN to the real grid. To evaluate the deep RL control performance, the proposed method was compared with a classic proportional control of the voltage change according to the power reference setup. The solution was validated in setups with a different number of thermostatically controlled loads (TCLs) in a feeder to show its generalization capabilities. In this article, the particularities of deep reinforcement learning application in the power system domain are discussed along with the results achieved by such an RL-powered demand response solution. The tuning of hyperparameters for the RL algorithm was performed to achieve the best performance of the double deep Q-network (DDQN) algorithm. In particular, the influence of a learning rate, a target network update step, network hidden layer size, batch size, and replay buffer size were assessed. The achieved performance is roughly two times better than the competing approach of optimal control selection within the considered time interval of the simulation. The decrease in deviation of the actual power consumption from the reference power profile is demonstrated. The benefit in costs is estimated for the presented voltage control-based ancillary service to show the potential impact.

Intelligent Voltage Control Method in Active Distribution Networks Based on Averaged Weighted Double Deep Q-network Algorithm

“…The calculations of SCC, SCP, SDC, and SDP are shown in ( 17)- (20). The SCC, SDC, SCP, or SDP of an EVA is the sum of the SCC, SDC, SCP, or SDP of EVs connected to this aggregator at time t, as in ( 21)- (24).…”

“…where N lt is the total number of EVs connected to the l th EVA at time t. Equations ( 17) - (24) give a quantitative representation of the schedulable capacity and power range of EVs to participate in the voltage control in ADN without affecting the future use of EVs [28], [29]. The maximum adjustable active power of an EVA is the value of its SCP, and the minimum adjustable active power is the negative value of its SCP, i.e., P jt,max = SCP jt and P jtmin = -SDP jt .…”

“…This algorithm prevents the overestimation of the action reward value by separating the selection and evaluation of actions when designing the reward target, which has better performance than DQN in Atari 2600 game tests. The voltage control with DDQN algorithm is found to be more effective than conventional proportional control [24]. However, DDQN algorithm also poses the problem of underestimating action reward values, especially in learning environments with stochastic relationships [25].…”

“…In contrast, AWDDQN algorithm can overcome the shortcomings of DQN and DDQN algorithms. However, there are few applications of DDQN and AWDDQN algorithms in the field of voltage control in ADNs [24].…”

Deep-Reinforcement-Learning-Based Two-Timescale Voltage Control for Distribution Systems

“…In [14], a DQN is used to control shunt capacitors. In [15], a double DQN is applied to achieve the optimal control of thermostatically controlled loads (TCLs) to provide voltage-control-based ancillary services. In [16], a multi-agent DQNbased algorithm is put forward to control switchable capacitors, voltage regulators, and smart inverters, where the continuous variables of voltage regulators and smart inverters are discretized.…”