Online Optimal Power Scheduling of a Microgrid via Imitation Learning

Gao, Shuhua; Chen, Xiang; Yu, Ming; Tan, K. T.; Lee, Tong Heng

doi:10.1109/tsg.2021.3122570

Cited by 55 publications

(16 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent efforts on the application of RL for energy management are reported in economic dispatch and optimal power flow [52,53], MG operation [54][55][56][57][58][59], and VPP [60].…”

Section: Energy Managementmentioning

confidence: 99%

“…For general energy management of MG or VPP [54,55,60], the reward is to minimize the operation cost, including the cost of DGs, energy storage, power exchange with grid, and controllable load. The action vector is the control signal of these components, and the state contains the control target and parameters or state of these components.…”

Section: Energy Managementmentioning

confidence: 99%

“…Typical reward and action can be expressed in equations ( 12) and ( 13): In [54], a model-based RL combining MCTS is used to conceive an online optimization agent that can teach itself how to optimally operate the MG through self-play. In [55], imitation learning is used to operate energy storage devices, converting the multi-step MILP into a simpler single-step MILP. In [60], offline simulator-based RL training and online practical environment-based training are combined to ameliorate the algorithm performance of the start-up process with prior knowledge.…”

Section: Energy Managementmentioning

confidence: 99%

See 2 more Smart Citations

Application of reinforcement learning in planning and operation of new power system towards carbon peaking and neutrality

Sun¹,

Wang²,

Huang³

et al. 2023

Prog. Energy

View full text Add to dashboard Cite

To mitigate global climate change and ensure a sustainable energy future, China has launched a new energy policy of achieving carbon peaking by 2030 and carbon neutrality by 2060, which sets an ambitious goal of building New Power System (NPS) with high penetration of renewable energy. However, the strong uncertainty, nonlinearity, and intermittency of renewable generation and their power electronics-based control devices are imposing grand challenges for secure and economic planning and operation of the NPS. The performance of traditional methods and tools becomes rather limited under such phenomena. Together with high-fidelity modeling and high-performance simulation techniques, the fast development of artificial intelligence (AI) technology, especially reinforcement learning (RL), provides a promising way of tackling these critical issues. This paper first provides a comprehensive overview of RL methods that interact with high-fidelity grid simulators to train effective agents for intelligent, model-free decision-making. Secondly, three important applications of RL are reviewed, including device-level control, system-level optimized control, and demand side management, with detailed modeling and procedures of solution explained. Finally, this paper discusses future research efforts for achieving the goals of full absorption of renewable energy, optimized allocation of large-scale energy resources, reliable supply of electricity, and secure and economic operation of the power grid.

show abstract

“…Recent efforts on the application of RL for energy management are reported in economic dispatch and optimal power flow [52,53], MG operation [54][55][56][57][58][59], and VPP [60].…”

Section: Energy Managementmentioning

confidence: 99%

Section: Energy Managementmentioning

confidence: 99%

Section: Energy Managementmentioning

confidence: 99%

See 1 more Smart Citation

Application of reinforcement learning in planning and operation of new power system towards carbon peaking and neutrality

Sun¹,

Wang²,

Huang³

et al. 2023

Prog. Energy

View full text Add to dashboard Cite

show abstract

“…Hua recently reviewed the use of imitation and transfer learning in the field of robotics, especially for tasks such as manipulation and fine control [2]. Recently, a method to use imitation learning in the context of a building energy management problem was described by Gao et al [1], where the final agent learns to imitate the actions of a Model Predictive Controller (MPC) with a perfect forecast of the environment variables. Pezzotti et al developed MimicBot, a deep RL agent in a Fantasy Football game environment, which achieved excellent performance when RL training techniques were applied after an initial behaviour cloning step [9].…”

Section: Related Workmentioning

confidence: 99%

Behavioural cloning based RL agents for district energy management

Kumar

Easwaran

Delinchant

et al. 2022

Proceedings of the 9th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

View full text Add to dashboard Cite

In this work, we discuss a method to incorporate domain knowledge into a Reinforcement Learning (RL) environment through the process of behavioral cloning, in the context of a district energy management system. Prior knowledge, encoded into heuristic rule-based programs, is used to initialize a policy network for an RL agent, after which an RL algorithm is used to improve on this by optimizing against a reward function. The key ideas are implemented in the CityLearn framework, where the resulting controller is used to manage the electrical energy storage for 5 buildings in a district. We demonstrate that the resulting agents offer measurable performance gains compared to existing baselines, offering a 3.8% improvement over the underlying rule-based controller, and a 20% improvement over a pure RL based controller. We also illustrate the possibility of using imitation learning to develop agents with desirable characteristics without explicit reward shaping. CCS CONCEPTS• Computing methodologies → Intelligent agents.

show abstract

“…However, IL has the issue of distributional shift and tends to have poor generalization ability [25]. Moreover, IL is dependent on expert experience and prone to error, especially in off-course situations where lack of corresponding demonstrations makes the errors intensify as trajectory steps accumulate [26] while DRL will not suffer from this problem since it continuously interacts with the environment and has self-improvement ability.…”

Section: Introductionmentioning

confidence: 99%

Digital twin‐based online resilience scheduling for microgrids: An approach combining imitative learning and deep reinforcement learning

Sun,

Liu,

Tan

et al. 2023

IET Renewable Power Gen

View full text Add to dashboard Cite

Strong uncertainty of renewables puts high demands on the fast response of flexibility resources and resilience‐oriented optimal scheduling for microgrids (MGs). Digital twins (DT) technology based on data‐driven methods is a potential solution to this problem. A DT‐based online resilience scheduling framework for MGs is designed in this study. Based on the proposed framework, a hybrid sequential‐parallel combination method of imitation learning (IL) and deep reinforcement learning (DRL) is proposed to develop the optimal scheduling strategy for MGs. First, a mixed integer second‐order cone programming (MISOCP) model is adopted to behave as an expert to generate decision demonstrations corresponding to operation scenarios of MGs. Then, IL and deep deterministic policy gradient (DDPG) are combined by (1) sequential pretrain and finetune and (2) parallel experience storing and sampling two steps to learn the optimal policy for MGs. Expert demonstrations from the MISOCP model are utilized to pretrain a deep neural network model by IL and initialize the policy network of DDPG to avoid it learning from scratch. Moreover, an expert replay buffer is introduced specifically to avoid forgetting the well‐behaved expert experience and to further accelerate the training process. A 6‐bus MG test system case study demonstrates the effectiveness and scalability of the proposed approach.

show abstract

Online Optimal Power Scheduling of a Microgrid via Imitation Learning

Cited by 55 publications

References 38 publications

Application of reinforcement learning in planning and operation of new power system towards carbon peaking and neutrality

Application of reinforcement learning in planning and operation of new power system towards carbon peaking and neutrality

Behavioural cloning based RL agents for district energy management

Digital twin‐based online resilience scheduling for microgrids: An approach combining imitative learning and deep reinforcement learning

Contact Info

Product

Resources

About