Building Energy Management With Reinforcement Learning and Model Predictive Control: A Survey

Abstract

Building energy management has been recognized as one of the core problems in modern power grids with respect to system operation efficiency. However, the building energy management system (BEMS) is now facing more challenges and uncertainties with the increasing penetration of renewable energy and increasing adoption of different types of electrical appliances and equipment. Classical model predictive control (MPC) has shown effective in building energy management, although it suffers from labour-intensive modelling and complex online control optimization. Recently, with the growing accessibility to building control and automation data, data-driven solutions such as data-driven MPC and reinforcement learning (RL)-based methods have attracted more research interest. However, the potential of integrating these two types of methods and how to choose suitable control algorithms have not been well discussed. In this work, we first present a compact review of the recent advances in data-driven MPC and RL-based control methods for building energy management. Furthermore, the main challenges in these approaches and general discussions on the selection of control methods are discussed.INDEX TERMS Building energy management, model predictive control, reinforcement learning, datadriven control.

A Systematic Literature Review on Data-Driven Residential and Industrial Energy Management Systems

“…Real-time semantic feature selection approaches, unsupervised temporal energy pattern characterization, multi-agent systems, generalized automated DR [8] Coordinated home EMS: topologies, techniques Cooperative learning, robust coordination, federated reinforcement learning (RL), uncertainties, blockchain technology, and privacy [9] Data-driven predictive control for demand-side management Robust feature selection, benchmark dataset, data quality, transferrable and scalable data-models [10] Home EMS with appliances and scheduling Grid reliability, load, and RES coordination, security, and privacy [11] Residential flexibility Standardized representation of flexibility resources, quantifying energy flexibility [12] Home EMS with concepts, architecture, infrastructure Appliance diversity, multi-objectives, consumption uncertainty, real-time interaction, continuous updating of feedback [13] Home EMS: DR, scheduling, optimization, and communication Self-learning systems to minimize user involvement [14] Home EMS: concepts, configuration, and technologies - [15] RL and model predictive control Data efficiency, safety, generalization, and robust adaptability [16] Building load prediction Algorithm development, feature selection, extraction, clustering, forecasting [3] EMS for smart grids considering user behavior Greenhouse gas (GHG) emissions, DR participation, data security and privacy, customer awareness, outdated infrastructure, highly uncertain systems [17] Residential demand-side management, optimization Include risk minimization in optimization, highly uncertain systems, standardized load classification, and other objectives, such as frequency/voltage stability [18] Multi-level EMS: architecture, objectives Smart transformer, reactive power in EMS, an uncertainty factor of EVs [19] Microgrids: control methods - [20] Microgrids: control and optimization methods -…”

“…They identify six challenges: handling GHG emissions, DR participation, data security and privacy, customer awareness and participation, outdated infrastructure, and highly uncertain systems. Zhang et al [16] focus on data-driven model predictive control and RL-based control algorithms for building EMSs. Identified future challenges for model predictive control are design complexity, model dependency, and time-consuming computations, while research on RL could tackle data efficiency, safety, generalization, and robust adaptability problems.…”

A Learning-Based Model Predictive Control Strategy for Home Energy Management Systems

“…DMPC (also known as learning-based MPC (LBMPC)) combines the advantages of machine learning (ML) and MPC, which can effectively mitigate the effects of disturbances and uncertainties. DMPC for HEMS can be divided into two main categories [37]. The first category (noted as Type I) trains predictive models offline from data with uncertainty • Exploit the thermodynamic model of the house to predict its behavior.…”

Reinforcement Learning: Theory and Applications in HEMS

“…A more recent review on building energy management [42] focuses on deep neural networks based RL. A recent article [43] considers RL along with model predictive control in smart building applications. The article in [44] is a survey of RL in demand response.…”