Model-predictive control and reinforcement learning in multi-energy system case studies

Abstract

Model-predictive-control (MPC) offers an optimal control technique to establish and ensure that the total operation cost of multi-energy systems remains at a minimum while fulfilling all system constraints. However, this method presumes an adequate model of the underlying system dynamics, which is prone to modelling errors and is not necessarily adaptive. This has an associated initial and ongoing project-specific engineering cost. In this paper, we present an on-and off-policy multi-objective reinforcement learning (RL) approach, that does not assume a model a priori, benchmarking this against a linear MPC (LMPC -to reflect current practice, though non-linear MPC performs better) -both derived from the general optimal control problem, highlighting their differences and similarities. In a simple multi-energy system (MES) configuration case study, we show that a twin delayed deep deterministic policy gradient (TD3) RL agent offers potential to match and outperform the perfect foresight LMPC benchmark (101.5%). This while the realistic LMPC, i.e. imperfect predictions, only achieves 98%. While in a more complex MES system configuration, the RL agent's performance is generally lower (94.6%), yet still better than the realistic LMPC (88.9%). In both case studies, the RL agents outperformed the realistic LMPC after a training period of 2 years using quarterly interactions with the environment. We conclude that reinforcement learning is a viable optimal control technique for multi-energy systems given adequate constraint handling and pre-training, to avoid unsafe interactions and long training periods, as is proposed in fundamental future work.

Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges

“…In [108], the management of IESs with integrated demand response is modeled as a Stackelberg game, and an actor-critic scheme is developed for the energy provider to adjust pricing and power dispatching strategies to cope with unknown private parameters of users. Extensive case studies are conducted in [109] to compare the performance of a twin delayed DDPG scheme against a benchmark linear model-predictive-control method, which empirically show that RL is a viable optimal control technique for IES management and can outperform conventional approaches.…”

Physics informed neural networks for control oriented thermal modeling of buildings

“…Works such as [7,23] have studied RL-based controllers in the context of building control and show that such RL controllers can lead to 5 − 12% energy savings compared to the existing rule-based controllers. Additionally, [24] compares the performance of MPC and RL controllers to show that RL controllers are able to outperform a linear MPC-based controller for two different test scenarios. Though these works indicate promising results for RL-based controllers, they also highlight existing challenges in real-world deployment of RL.…”

Physics Informed Neural Networks for Control Oriented Thermal Modeling of Buildings

“…Works such as [18,19] have studied RL-based controllers in the context of building control using different RL algorithms such as deep Q-networks, fitted Q-iteration and show that such RL controllers can lead to 5 − 12% energy savings compared to the existing rule-based controllers. Additionally, [20] compares the performance of MPC and RL controllers. The authors show that RL controllers are able to outperform a linear MPC-based controller for two different test scenarios.…”