Attention and masking embedded ensemble reinforcement learning for smart energy optimization and risk evaluation under uncertainties

Abstract

Integrating residential-level photovoltaic energy generation and energy storage for the on-grid system are essential to reduce electricity use for residential consumption from the grid. However, reaching a reliable and optimal control policy is highly challenging due to the intrinsic uncertainties in the renewable energy sources and fluctuating demand profile. In this work, we proposed and designed an ensemble deep reinforcement learning (DRL) algorithm combined with risk evaluation to solve the energy optimization problem under uncertainties. Meanwhile, the attention and masking layer, the state-of-the-art natural language processing techniques, were incorporated into the algorithm to handle the issue of hard constraints, which are frequently encountered in the renewable energy optimization problem. To the best of our knowledge, this work is the first attempt to tackle the energy optimization problem under uncertainty using a scenario-based ensemble DRL approach with a risk evaluation. Through a well-designed single household microgrid energy management system we found that the attention and masking layer played a crucial role in fulfilling the hard constraint. The ensemble DRL with the increased number of agents showed a significantly improved energy management policy leading to ~75% of the cost reduction compared with those obtained by using conventional DRL with a single agent. The risk evaluation revealed that the current ensemble DRL approach possessed a high risk/high profit feature, which could be significantly improved by designing a risk-aware reward function in future investigations.

Ensemble Reinforcement Learning: A Survey

“…Network: TCN, BiLSTM, KELM, BPNN, MLP, echo state network (ESN), Elman neural network (ENN), and gradient boosting decision tree (GBDT); Training algorithm: SARSA Qin et al [85] historical load data of the California Independent System Operator (CASIO) from January 1, 2021 to July 5, 2021 PPO guided tree search, the MIQP algorithm with Gurobi 9.1 Sogabe et al [86] optimal energy management in a residential building microgrid mixed-integer linear programming (MILP) Birman et al [60] a range of real-world scenarios Aggregation method Li et al [51] PHEME, RumorEval Naive Bayes, SVM-SGD, Dense, BiLSTM, FastText, TextCNN, VRoC, some combinations of above methods Sharma et al [52] two MEC servers and 30 IoTDs randomly distributed in the squared area with size 50m×50m…”

“…Compared with traditional studies using machine learning (ML) or artificial neural network (ANN) prediction methods [79,80], there is a significant gap between the prediction results obtained by these methods and ERL. Liu [83] wind speed prediction deep Q-Network 2021 Jalali et al [84] wind power forecasting Q-learning 2022 Tan et al [31] PM2.5 prediction Sarsa 2022 Qin et al [85] unit commitment problem deep Q-Network 2022 Sogabe et al [86] smart energy optimization and risk evaluation Q-learning 2022 Sharma et al [52] estimating reference evapotranspiration Q-learning 2022 He et al [87] wind farm control deep deterministic policy gradient 2022 Jalali et al [88] solar irradiance forecasting Q-learning 2022 Shi Yin and Hui Liu [61] wind power prediction Q-learning 2023 Yu et al [29] wind power prediction deep deterministic policy gradient…”

AI-Assisted Decision-Making and Risk Evaluation in Uncertain Environment Using Stochastic Inverse Reinforcement Learning: American Football as a Case Study

“…Here, T(s t+1 |s t , a t ) is the stochastic transition probability and will be given in more detail in Section 2.4. The integrated final visiting frequency of specified state s i in P(ξ i |θ) under a given trajectory length T for all s t is given as follows and will be used in formula (17) to update the learning parameter θ.…”

“…It is a conventional approach used in financial engineering field. Another one is the test demonstration of the trained model on the data that are not included in the training data, which is a standard measure used in the field of machine learning [17].…”

“…For example, virtual game RL programs, such as AlphaGo [21], AlphaGo Zero [22], and Atari [23], have defeated human players. Moreover, RL has also been applied to practical sports, such as Ice Hockey [24], Soccer [25,26], Basketball [27], and smart grid energy optimization [17,28]. Various IRL algorithms have been proposed so far [29] for various applications, such as inferring and optimizing the strategy of offensive and defensive play in soccer game [30], optimizing the brushstroke quality in computer-generated painting [31], learning the car-following behavior of drivers [32], programming by demonstrations to develop advanced learning systems [33], and boosting table-to-text generation [34,35].…”