Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks

Gao, Yuanqi; Wang, Wei; Yu, Nanpeng

doi:10.1109/tsg.2021.3058996

Cited by 111 publications

(49 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of VVC, existing studies were mainly focused on various aspects of scaling RL to the challenges specific to the VVC, such as minimizing constraint violations [Wang et al, 2020b] and scaling to combinatorially large actions spaces [Zhang et al, 2021]. Alternatively, researchers have also tackled the VVC problem by formulating it as multi-agent reinforcement learning (MARL) problem and proposing a novel efficient and resilient MARL algorithm [Gao et al, 2021]. Additionally, a more recent and closely related work in terms of methodology by [Zhao and Wang, 2021] also proposed to combine RL with graph neural networks for power system restoration via a multi-agent formulation.…”

Section: Related Workmentioning

confidence: 99%

A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systems

Lee¹,

Sarkar²,

Wang³

2021

Preprint

View full text Add to dashboard Cite

Volt-var control (VVC) is the problem of operating power distribution systems within healthy regimes by controlling actuators in power systems. Existing works have mostly adopted the conventional routine of representing the power systems (a graph with tree topology) as vectors to train deep reinforcement learning (RL) policies. We propose a framework that combines RL with graph neural networks and study the benefits and limitations of graph-based policy in the VVC setting. Our results show that graph-based policies converge to the same rewards asymptotically however at a slower rate when compared to vector representation counterpart. We conduct further analysis on the impact of both observations and actions: on the observation end, we examine the robustness of graph-based policy on two typical data acquisition errors in power systems, namely sensor communication failure and measurement misalignment. On the action end, we show that actuators have various impacts on the system, thus using a graph representation induced by power systems topology may not be the optimal choice. In the end, we conduct a case study to demonstrate that the choice of readout function architecture and graph augmentation can further improve training performance and robustness.

show abstract

Section: Related Workmentioning

confidence: 99%

A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systems

Lee¹,

Sarkar²,

Wang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…complex systems, including games [5] and autonomous driving [6]. Recently, MARL approaches have also found applications in the power systems domain, with an emphasis on voltage regulation problems [7]- [9]. These applications utilize the capabilities of MARL to devise local control policies without any knowledge of the models of the underlying complex systems.…”

Section: A Multi-agent Reinforcement Learning In Energy Systemsmentioning

confidence: 99%

PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems

Biagioni¹,

Zhang²,

Wald³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present the PowerGridworld software package to provide users with a lightweight, modular, and customizable framework for creating power-systems-focused, multi-agent Gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). Although many frameworks exist for training multi-agent RL (MARL) policies, none can rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multidevice) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an opensource software package that helps to fill this gap. To highlight PowerGridworld's key features, we present two case studies and demonstrate learning MARL policies using both OpenAI's multi-agent deep deterministic policy gradient (MADDPG) and RLLib's proximal policy optimization (PPO) algorithms. In both cases, at least some subset of agents incorporates elements of the power flow solution at each time step as part of their reward (negative cost) structures.

show abstract

“…In distribution systems, voltage profiles are the most critical indicator of the system operating condition, whilst reliable and efficient energy management is the core task [1][2][3][4]. This is why Volt-VAR control (VVC) schemes have been developed and integrated into distribution systems to reduce network losses [2], avoid voltage violations [5] and mitigate cyber attacks [6]. However, the rapid growth of distributed energy resources makes it increasingly difficult to manage voltage profiles on active distribution networks.…”

Section: Introduction a Background And Motivationmentioning

confidence: 99%

Deep Reinforcement Learning with Graph ConvNets for Distribution Network Voltage Control

Wu¹,

Carreño²,

Scaglione³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper proposes a model-free Volt-VAR control (VVC) algorithm via the spatio-temporal graph ConvNet-based deep reinforcement learning (STGCN-DRL) framework, whose goal is to control smart inverters in an unbalanced distribution system. We first identify the graph shift operator (GSO) based on the power flow equations. Then, we develop a spatiotemporal graph ConvNet (STGCN), testing both recurrent graph ConvNets (RGCN) and convolutional graph ConvNets (CGCN) architectures, aimed at capturing the spatiotemporal correlation of voltage phasors. The STGCN layer performs the feature extraction task for the policy function and the value function of the reinforcement learning architecture, and then we utilize the proximal policy optimization (PPO) to search the action spaces for an optimum policy function and to approximate an optimum value function. We further utilize the low-pass property of voltage graph signal to introduce an GCN architecture for the the policy whose input is a decimated state vector, i.e. a partial observation. Case studies on the unbalanced 123-bus systems validate the excellent performance of the proposed method in mitigating instabilities and maintaining nodal voltage profiles within a desirable range.

show abstract

Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks

Cited by 111 publications

References 38 publications

A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systems

A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systems

PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems

Deep Reinforcement Learning with Graph ConvNets for Distribution Network Voltage Control

Contact Info

Product

Resources

About