Building HVAC Scheduling Using Reinforcement Learning via Neural Network Based Model Approximation

Abstract

Buildings sector is one of the major consumers of energy in the United States. The buildings HVAC (Heating, Ventilation, and Air Conditioning) systems, whose functionality is to maintain thermal comfort and indoor air quality (IAQ), account for almost half of the energy consumed by the buildings. Thus, intelligent scheduling of the building HVAC system has the potential for tremendous energy and cost savings while ensuring that the control objectives (thermal comfort, air quality) are satisfied.Traditionally, rule-based and model-based approaches such as linear-quadratic regulator (LQR) have been used for scheduling HVAC. However, the system complexity of HVAC and the dynamism in the building environment limit the accuracy, efficiency and robustness of such methods. Recently, several works have focused on model-free deep reinforcement learning based techniques such as Deep Q-Network (DQN). Such methods require extensive interactions with the environment. Thus, they are impractical to implement in real systems due to low sample efficiency. Safetyaware exploration is another challenge in real systems since certain actions at particular states may result in catastrophic outcomes.To address these issues and challenges, we propose a modelbased reinforcement learning approach that learns the system dynamics using a neural network. Then, we adopt Model Predictive Control (MPC) using the learned system dynamics to perform control with random-sampling shooting method. To ensure safe exploration, we limit the actions within safe range and the maximum absolute change of actions according to prior knowledge. We evaluate our ideas through simulation using widely adopted EnergyPlus tool on a case study consisting of a two zone data-center. Experiments show that the average deviation of the trajectories sampled from the learned dynamics and the ground truth is below 20%. Compared with baseline approaches, we reduce the total energy consumption by 17.1% ∼ 21.8%. Compared with model-free reinforcement learning approach, we reduce the required number of training steps to converge by 10x.The energy consumption by buildings consist of 40% of the total energy and 70% of total electricity in the United States [18]. Of the total energy consumption of buildings, the Heating, Ventilation and Air-Conditioning (HVAC) system accounts for 50% while the rest is used for lighting, electrical appliances, electric vehicles, etc. The main objective of the HVAC system is to maintain the indoor temperature and air quality. An intelligent HVAC scheduling system will, additionally, save energy and cost while satisfying the objective. The HVAC system is a nonlinear system and has complex system dynamics with a large number of subsystems including chillers, boilers, heat pumps, pipes, ducts, fans, pumps and heat exchangers [11]. In this paper, we assume the combination of equipment to operate by the HVAC system is fixed and focus on how to set the temperature point for local controllers to reduce the energy arXiv:1910.05313v1 [eess.SY]

Indoor air quality prediction systems for smart environments: A systematic review

“…In this list, two papers only focus on outdoor air quality [64,97], eight papers do not include any AI-specific prediction algorithms [22,31,47,50,84,91,116,122] or were based on some mathematical approaches. Three papers [12,96,137] only focused on thermal comfort (temperature and/or humidity data) or other smart building aspects instead of air quality. Moreover, two studies [89,120] were rejected because they were limited to a monitoring system design and no prediction system was implemented.…”

Energy Management of Smart Home with Home Appliances, Energy Storage System and Electric Vehicle: A Hierarchical Deep Reinforcement Learning Approach

“…A holistic DRL method for the energy management of commercial buildings was presented in [30] where Heating, Ventilation, and Air conditioning (HVAC) system, lighting, blind, and window systems are controlled to achieve energy savings within the buildings' occupants comfort in terms of thermal, air quality, and illumination conditions. To resolve the limit of model-free DRL methods such as low sample efficiency, a model-based RL method was developed for building HVAC control that trains the system dynamics using neural networks [31]. Based on the trained system dynamics, the operation of the HVAC system was managed by model predictive control to minimize both the energy cost and the indoor temperature constraints violation.…”

Mb2c

“…Although a simulated building model can be used to accelerate the training process, it needs a highdelity model, which is hard to calibrate [6,7]. Recently, Model-Based Reinforcement Learning (MBRL) has been tested for HVAC control to achieve high data eciency [10]. The HVAC system BuildSys '20, November 18-20, 2020, Virtual Event, Japan Xianzhong Ding, Wan Du, and Alberto E. Cerpa dynamics is rst learned using a neural network based on historical HVAC data.…”

“…The HVAC system BuildSys '20, November 18-20, 2020, Virtual Event, Japan Xianzhong Ding, Wan Du, and Alberto E. Cerpa dynamics is rst learned using a neural network based on historical HVAC data. Based on the learned building dynamics model, an MPC controller tries to nd the optimal control action by using a Random Shooting (RS) method [10]. For controlling a single-zone HVAC system, an MBRL-based approach saves approximately 10⇥ training time of the MFRL approach, while achieving comparable performance [10].…”

“…Based on the learned building dynamics model, an MPC controller tries to nd the optimal control action by using a Random Shooting (RS) method [10]. For controlling a single-zone HVAC system, an MBRL-based approach saves approximately 10⇥ training time of the MFRL approach, while achieving comparable performance [10]. However, most of the commercial buildings are multi-zone buildings [11].…”