On the Joint Control of Multiple Building Systems with Reinforcement Learning

Zhang, Tianyu; Baasch, Gaby; Ardakanian, Omid; Evins, Ralph

doi:10.1145/3447555.3464855

Cited by 8 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Learning-based control has immense potential for energy savings and improved IEQ. In recent years, various types of model-free RL have been applied to the HVAC control problem with the goal of finding a near-optimal control policy without modelling the complex building dynamics [42,48]. In this approach, a mapping between the building state and optimal control is learned through interaction with the building.…”

Section: Building Hvac Controlmentioning

confidence: 99%

“…In this approach, a mapping between the building state and optimal control is learned through interaction with the building. It is shown in [48] that a single RL agent that observes the state of the whole building and controls all setpoints can reduce the total HVAC energy consumption by around 22% compared to a rule-based controller in a small multizone building. However, learning such a control policy requires extensive interaction with the environment and over 300 months of sensor data to fully explore the state and action space.…”

Section: Building Hvac Controlmentioning

confidence: 99%

“…Identifying this model from the sensor data that is available through the BMS or gathered by smart thermostats [17] is nontrivial in large multizone buildings due to limited observability and lack of sufficient excitation [3]. Model-free RL techniques, on the other hand, learn a control policy by directly interacting with the building environment and systems [48]. Given the size of the state and action space, training an optimal policy requires many interaction episodes in large multi-zone buildings.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Diversity for transfer in learning-based control of buildings

Zhang

Afshari

et al. 2022

Proceedings of the Thirteenth ACM International Conference on Future Energy Systems

Self Cite

View full text Add to dashboard Cite

The application of reinforcement learning to the optimal control of building systems has gained traction in recent years as it can cut the building energy consumption and improve human comfort. Despite using sample-efficient reinforcement learning algorithms, most related work requires several months of sensor data and operational parameters of the building to train an agent that outperforms existing rule-based controllers in a large multi-zone building. Moreover, exploring the large state and action spaces can result in poor indoor environmental quality for occupants. In this paper, we propose to reduce the training cost of a policy gradient reinforcement learning algorithm by learning a library of control policies on a training building and taking advantage of both environmental and policy diversity. To transfer these policies to a target building, which can be different from the training building, we develop a simple method to assign the best pretrained policy in the library to each zone of the target building. We show that even without retraining the transferred policies on the target building, they can reduce the HVAC energy consumption by 40.4% compared to a fixed-schedule baseline and by 48.97% compared to agents trained on the target building for 5,000 months. The plausibility of our results underscores the importance of using diversity and transfer learning in multi-agent reinforcement learning settings and could pave the way for the adoption of reinforcement-learning based controllers in real buildings.

show abstract

Section: Building Hvac Controlmentioning

confidence: 99%

Section: Building Hvac Controlmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diversity for transfer in learning-based control of buildings

Zhang

Afshari

et al. 2022

Proceedings of the Thirteenth ACM International Conference on Future Energy Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Open data is critical for validating improved DRL methods in the context of BEMS, and open codes can aid in reproducing such results as well as accelerating the progress of this research area similar to the previously mentioned CityLearn environment. Out of the large data base analyzed, very few researchers have notably open sourced and shared their codes such as Zhang et al [95,106], Touzani et al [99], and Svetozarevic et al [25], while Marzullo et al introduced a full open-source simulation environment for advanced building control performance testing [91]. Meanwhile a few other researchers noted the availability of their data on request [69,84,88,89].…”

Section: Citylearn: a Multi-agent Rl Environment For Large-scale Bems...mentioning

confidence: 99%

Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types

Shaqour

Hagishima

2022

Energies

View full text Add to dashboard Cite

Owing to the high energy demand of buildings, which accounted for 36% of the global share in 2020, they are one of the core targets for energy-efficiency research and regulations. Hence, coupled with the increasing complexity of decentralized power grids and high renewable energy penetration, the inception of smart buildings is becoming increasingly urgent. Data-driven building energy management systems (BEMS) based on deep reinforcement learning (DRL) have attracted significant research interest, particularly in recent years, primarily owing to their ability to overcome many of the challenges faced by conventional control methods related to real-time building modelling, multi-objective optimization, and the generalization of BEMS for efficient wide deployment. A PRISMA-based systematic assessment of a large database of 470 papers was conducted to review recent advancements in DRL-based BEMS for different building types, their research directions, and knowledge gaps. Five building types were identified: residential, offices, educational, data centres, and other commercial buildings. Their comparative analysis was conducted based on the types of appliances and systems controlled by the BEMS, renewable energy integration, DR, and unique system objectives other than energy, such as cost, and comfort. Moreover, it is worth considering that only approximately 11% of the recent research considers real system implementations.

show abstract

“…Sequence-to-sequence models [6][7][8] and Bayesian networks [16,28] were applied to make predictions in model predictive control. Soft actor-critic (SAC) [10] is chosen in this work due to its promising results in energy management [4,21,30,40].…”

Section: Introductionmentioning

confidence: 99%

Addressing partial observability in reinforcement learning for energy management

Biemann

Liu

Zeng

et al. 2021

Proceedings of the 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

View full text Add to dashboard Cite

Automatic control of energy systems is affected by the uncertainties of multiple factors, including weather, prices and human activities. The literature relies on Markov-based control, taking only into account the current state. This impacts control performance, as previous states give additional context for decision making. We present two ways to learn non-Markovian policies, based on recurrent neural networks and variational inference. We evaluate the methods on a simulated data centre HVAC control task. The results show that the off-policy stochastic latent actor-critic algorithm can maintain the temperature in the predefined range within three months of training without prior knowledge while reducing energy consumption compared to Markovian policies by more than 5%. CCS CONCEPTS• Computing methodologies → Reinforcement learning; • Mathematics of computing → Markov processes; Variational methods.

show abstract

On the Joint Control of Multiple Building Systems with Reinforcement Learning

Cited by 8 publications

References 17 publications

Diversity for transfer in learning-based control of buildings

Diversity for transfer in learning-based control of buildings

Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types

Addressing partial observability in reinforcement learning for energy management

Contact Info

Product

Resources

About