Application of deep Q-networks for model-free optimal control balancing between different HVAC systems

Abstract

A deep Q-network (DQN) was applied for model-free optimal control balancing between different HVAC systems. The DQN was coupled to a reference office building: an EnergyPlus simulation model provided by the U.S. Department of Energy. The building was air-conditioned with four air-handling units (AHUs), two electric chillers, a cooling tower, and two pumps. EnergyPlus simulation results for eleven days (July 1-11) and three subsequent days (July 12-14) were used to improve the DQN policy and test the optimal control. The optimization goal was to minimize the building's energy use while maintaining the indoor CO 2 concentration below 1,000 ppm. It was revealed that the DQN-a reinforcement learning method-can improve its control policy based on prior actions, states, and rewards. The DQN lowered the total energy usage by 15.7% in comparison with the baseline operation while maintaining the indoor CO 2 concentration below 1,000 ppm. Compared to model predictive control, the DQN does not require a simulation model, or a predetermined prediction horizon, thus delivering model-free optimal control. Furthermore, it was demonstrated that the DQN can find balanced control actions between different energy consumers in the building, such as chillers, pumps, and AHUs.

Evaluating the Adaptability of Reinforcement Learning Based HVAC Control for Residential Houses

“…In the past few years, reinforcement learning-based (RL) approaches were proposed for intelligent control of HVAC [15][16][17][18][19][20][21][22][23]. Some of these approaches such as [15][16][17] used a Q-learning algorithm-a model-free RL algorithm.…”

“…Very recently, a deep learning based reinforcement learning (DRL) approach has gotten a lot of traction for optimizing the HVAC control [18][19][20][21][22]. Wei et al [18] used the DRL approach, for the first time, for HVAC control.…”

“…The study concluded that, although the model-free RL methods under performed their model-based counterpart, they can provide substantial cost saving with significantly less computational cost as compared to the model-based control. In a similar recent work, Ahn and Park [21] used a DQN based model-free RL algorithm to obtain the control balance between different HVAC systems with the goal of minimizing building's energy usage while maintaining CO 2 concentration below 1000 ppm. The DQN showed a 15% reduction in the energy usage and also maintained the CO 2 level below 1000 ppm.…”

“…We found limited research on analyzing the performance of RL-based HVAC control models in a real house. Ahn and Park [21] highlighted the need of approaches such as transfer learning in applying the DQN to enhance their self-learning ability so that trained RL models (in simulated environment) can be utilized in real-world situations. At present, we found two research papers focusing on the practical implementation and deployment of the DRL-based HVAC control [22,25].…”

Solar+ Optimizer: A Model Predictive Control Optimization Platform for Grid Responsive Building Microgrids

“…A neural network model was trained using the Levenberg-Marquardt algorithm to predict the optimal boiler operation period in commercial buildings [34]. Another emerging type of advanced control is Deep Reinforcement Learning (DRL) [35][36][37][38][39][40]. While DRL is a model-free approach, most DRL methods use detailed models to generate synthetic data for training purposes.…”