Deep Multi-Agent Reinforcement Learning for Cost Efficient Distributed Load Frequency Control

Abstract

The rise of microgrid-based architectures is heavily modifying the energy control landscape in distribution systems making distributed control mechanisms necessary to ensure reliable power system operations. In this paper, we propose the use of Reinforcement Learning techniques to implement load frequency control without requiring a central authority. To this end, we approximate the optimal solution of the primary, secondary, and tertiary control with the use of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. Generation units are characterised as agents that learn how to maximise their long-term performance by acting and interacting with the environment to balance generation and load in a cost efficient way. Network effects are also modelled in our framework for the restoration of frequency to the nominal value. We validate our Reinforcement Learning methodology through numerical results and show that it can be used to implement the load frequency control in a distributed and cost efficient way.

Multiagent reinforcement learning for strictly constrained tasks based on Reward Recorder

“…For condition (b) in Theorem 2, suppose that at a certain time-step, an agent i in state s ˆi takes action a ˆi, with next state s ˆ′ i . The equivalent reward r ˆi is given in (13). We assume that this time step is the mth iteration of the state-action tuple s a s < ˆ, ˆ, ˆ′> i i i , where  m 1.…”

“…5,6 Outstanding research has achieved good evaluations in hybrid environments 7 or multiobjective tasks, 8 but it is only in recent years that MARL has been used in engineering applications. They mainly focus on scheduling and optimization problems in multiple engineering domains, such as traffic control, 9 autonomous driving, 10,11 base station communication, 12 load frequency optimization, 13 electric vehicle charging and discharging planning, 14 power allocation, 15 and so forth. In addition, as more complex problems are considered, more applicable systems are modeled and analyzed, such as Markov Repairable Systems, 16 Markov Jumping Systems, 17 and so forth.…”