Diversity for transfer in learning-based control of buildings

Abstract

The application of reinforcement learning to the optimal control of building systems has gained traction in recent years as it can cut the building energy consumption and improve human comfort. Despite using sample-efficient reinforcement learning algorithms, most related work requires several months of sensor data and operational parameters of the building to train an agent that outperforms existing rule-based controllers in a large multi-zone building. Moreover, exploring the large state and action spaces can result in poor indoor environmental quality for occupants. In this paper, we propose to reduce the training cost of a policy gradient reinforcement learning algorithm by learning a library of control policies on a training building and taking advantage of both environmental and policy diversity. To transfer these policies to a target building, which can be different from the training building, we develop a simple method to assign the best pretrained policy in the library to each zone of the target building. We show that even without retraining the transferred policies on the target building, they can reduce the HVAC energy consumption by 40.4% compared to a fixed-schedule baseline and by 48.97% compared to agents trained on the target building for 5,000 months. The plausibility of our results underscores the importance of using diversity and transfer learning in multi-agent reinforcement learning settings and could pave the way for the adoption of reinforcement-learning based controllers in real buildings.

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

“…This study demonstrated that effective transfer occurs when the DRL agent is moved between buildings situated in similar climatic conditions, mainly when two out of the five layers of the neural network approximating the control policy in the source building were shared between source and target control agents. Zhang et al (2022a) introduced a strategy aimed at transferring a multi-agent RL from a source building with multiple zones to a target building. In this work, the authors developed a methodology to choose the most suitable pre-trained RL policy obtained from the source building prior to the transfer learning process, managing zone temperature setpoints in a Variable Air Volume (VAV) system.…”

Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

“…This study is the only one in which the use of heterogeneous TL was evaluated since it assessed the possibility of transferring a control policy between buildings with different numbers of thermal zones. Zhang et al [54] implemented a strategy to transfer a library of RL multi-agent control policies from a multi-zone source building to a target building. Before transferring the control policy, the authors designed a strategy to choose the best pre-trained RL policy among those obtained on the source building for the management of zone temperature setpoints in a Variable Air Volume (VAV) system.…”

Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types

“…Similar to residential buildings, the concept of transfer learning has been explored recently and can be further expanded. Zhang et al reported that utilizing transfer learning for multi-agent DRL for multiple zone control of an HVAC system can reduce the energy consumption by 40.4% [83]. This improvement was observed even if the system policies originated in other buildings and were not locally retrained.…”