Deep Reinforcement Scheduling of Energy Storage Systems for Real-Time Voltage Regulation in Unbalanced LV Networks With High PV Penetration

Wang, Shengyi; Du, Liang; Fan, Xiaoyuan; Huang, Qiuhua

doi:10.1109/tste.2021.3092961

Cited by 41 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The RL-agent needs to be learnt so that the system operates for 100 time-steps without cascading. The learnable parameter, θ 1 , of the RL-agent is the value that maximizes the objective function (13), which is the sum of the rewards for each line. When an early termination occurs due to cascading, the rewards are only summed till the termination time-step.…”

Section: Discussionmentioning

confidence: 99%

“…Recently, with interest to develop real-time recommendation systems, artificial intelligence (AI) based controllers have been of interest to the industry [4], [5]. AI is used in diverse applications by the industry and few such examples are power grid voltage control [10], stability [11], emergency load shedding [12], energy storage systems [13]. [14] proposes a reinforcement learning-based topology controller but training such a controller to perform well over a wide range of operating scenarios is non-trivial.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading

Matavalam

Guddanti

Weng

et al. 2023

IEEE Trans. Power Syst.

View full text Add to dashboard Cite

This paper describes how domain knowledge of power system operators can be integrated into reinforcement learning (RL) frameworks to effectively learn agents that control the grid's topology to prevent thermal cascading. Typical RLbased topology controllers fail to perform well due to the large search/optimization space. Here, we propose an actor-critic-based agent to address the problem's combinatorial nature and train the agent using the RL environment developed by RTE, the French TSO. To address the challenge of the large optimization space, a curriculum-based approach with reward tuning is incorporated into the training procedure by modifying the environment using network physics for enhanced agent learning. Further, a parallel training approach on multiple scenarios is employed to avoid biasing the agent to a few scenarios and make it robust to the natural variability in grid operations. Without these modifications to the training procedure, the RL agent failed for most test scenarios, illustrating the importance of properly integrating domain knowledge of physical systems for real-world RL learning. The agent was tested by RTE for the 2019 learning to run the power network challenge and was awarded the 2 nd place in accuracy and 1 st place in speed. The developed code is open-sourced for public use. Analysis of a simple system proves the enhancement in training RL-agents using the curriculum.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading

Matavalam

Guddanti

Weng

et al. 2023

IEEE Trans. Power Syst.

View full text Add to dashboard Cite

show abstract

“…Market driven applications include energy arbitrage [128] and frequency reserves participation [152], [86], [161]. Other applications include PV generation peak shaving [82], loss minimization in distribution networks [150], mitigating voltage deviations in low voltage distribution networks with high PV penetration [157] and frequency instability reduction not related to frequency reserve market participation [108].…”

Section: B) Isolatedmentioning

confidence: 99%

Exploiting Battery Storages With Reinforcement Learning: A Review for Energy Professionals

2022

View full text Add to dashboard Cite

The transition to renewable production and smart grids is driving a massive investment to battery storages, and reinforcement learning (RL) has recently emerged as a potentially disruptive technology for their control and optimization of battery storage systems. A surge of papers has appeared in the last two years applying reinforcement learning to the optimization of battery storages in buildings, energy communities, energy harvesting Internet of Things networks, renewable generation, microgrids, electric vehicles and plug-in hybrid electric vehicles. This article reviews these applications through 4 different perspectives. Firstly, the type of optimization problem is analyzed; the literature can be divided to approaches that optimize either financial targets or energy efficiency. Secondly, the approaches for handling user comfort are analyzed for applications that may impact a human user. Thirdly, this paper discusses the approach to model and reduce battery degradation. Fourthly, the articles are categorized by application context and applications likely to attract a high amount of research are identified. The paper concludes with a list of unresolved challenges.

show abstract

“…Similar solutions are applicable to microgrids with local markets [62,63]. Only a few works consider frequency reserves [21][22][23] or ancillary services [64].…”

Section: Reinforcement Learning Applications For Batteriesmentioning

confidence: 99%

Bidding a Battery on Electricity Markets and Minimizing Battery Aging Costs: A Reinforcement Learning Approach

et al. 2022

View full text Add to dashboard Cite

Battery storage is emerging as a key component of intelligent green electricitiy systems. The battery is monetized through market participation, which usually involves bidding. Bidding is a multi-objective optimization problem, involving targets such as maximizing market compensation and minimizing penalties for failing to provide the service and costs for battery aging. In this article, battery participation is investigated on primary frequency reserve markets. Reinforcement learning is applied for the optimization. In previous research, only simplified formulations of battery aging have been used in the reinforcement learning formulation, so it is unclear how the optimizer would perform with a real battery. In this article, a physics-based battery aging model is used to assess the aging. The contribution of this article is a methodology involving a realistic battery simulation to assess the performance of the trained RL agent with respect to battery aging in order to inform the selection of the weighting of the aging term in the RL reward formula. The RL agent performs day-ahead bidding on the Finnish Frequency Containment Reserves for Normal Operation market, with the objective of maximizing market compensation, minimizing market penalties and minimizing aging costs.

show abstract

Deep Reinforcement Scheduling of Energy Storage Systems for Real-Time Voltage Regulation in Unbalanced LV Networks With High PV Penetration

Cited by 41 publications

References 29 publications

Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading

Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading

Exploiting Battery Storages With Reinforcement Learning: A Review for Energy Professionals

Bidding a Battery on Electricity Markets and Minimizing Battery Aging Costs: A Reinforcement Learning Approach

Contact Info

Product

Resources

About