Reinforcement Learning for Energy-Storage Systems in Grid-Connected Microgrids: An Investigation of Online vs. Offline Implementation

Abstract

Grid-connected microgrids consisting of renewable energy sources, battery storage, and load require an appropriate energy management system that controls the battery operation. Traditionally, the operation of the battery is optimised using 24 h of forecasted data of load demand and renewable energy sources (RES) generation using offline optimisation techniques, where the battery actions (charge/discharge/idle) are determined before the start of the day. Reinforcement Learning (RL) has recently been suggested as an alternative to these traditional techniques due to its ability to learn optimal policy online using real data. Two approaches of RL have been suggested in the literature viz. offline and online. In offline RL, the agent learns the optimum policy using predicted generation and load data. Once convergence is achieved, battery commands are dispatched in real time. This method is similar to traditional methods because it relies on forecasted data. In online RL, on the other hand, the agent learns the optimum policy by interacting with the system in real time using real data. This paper investigates the effectiveness of both the approaches. White Gaussian noise with different standard deviations was added to real data to create synthetic predicted data to validate the method. In the first approach, the predicted data were used by an offline RL algorithm. In the second approach, the online RL algorithm interacted with real streaming data in real time, and the agent was trained using real data. When the energy costs of the two approaches were compared, it was found that the online RL provides better results than the offline approach if the difference between real and predicted data is greater than 1.6%.

Dual-Layer Q-Learning Strategy for Energy Management of Battery Storage in Grid-Connected Microgrids

“…In this regard, the deviation between forecasted and actual (real) net demand by month is tracked and is 10%, 7%, −9%, −11%, 4.5%, 1.9%, −5%, −2%, −2.5%, −2%, 6.5%, and 2.3% from January to December. The convergence and cost savings of the proposed dual-layer Q-learning architecture are compared for a full year with those of the online and offline RL algorithms reported in [20]. The hyperparameters used in this work are γ, α, and ε.…”

“…The solution to deal with environmental uncertainties was presented in [18,19], which uses online RL methods to find the optimal control strategy for battery operation while interacting with the real system in real time. Our previous work in [20] has compared the performance of offline RL with that of online RL for managing ESS in microgrids. Synthetic forecasted data were constructed by adding white Gaussian noise with a range of standard deviations on real data to approximate forecasted data.…”

Design and Control of a Photovoltaic Distribution System Based on Modular Buck-Boost Converters

“…These studies proposed a novel approach that integrates machine learning techniques with power management algorithms to enhance the efficiency and performance of ASD. Reinforcement learning techniques proved to be effective through simulations and experimental validations, exhibiting significant improvements in energy management and system performance Ali et al (2021).…”

Real-Time Economic Dispatch of CHP Systems with Battery Energy Storage for Behind-the-Meter Applications

“…where γ CH bat and γ D bat are binary variables representing the "on/off" and "off" states of the BESS charge and discharge, respectively. The inequality constraints for charging and discharging the BESS are shown in Equation (18).…”

“…Combining CHP systems with battery storage can address these concerns; nevertheless, the dispatch of CHP and battery power must be managed to optimise the overall operation, with all generated power being consumed on-site. In an offline day-ahead optimisation, previous knowledge of the electrical load is essential, and thus, it will rely on it as forecasted load data [18]. The prediction error is likely to produce suboptimal dispatch commands that might result in power being injected into the grid, violating the 'behind-the-meter' constraints.…”