Near-optimal Deep Reinforcement Learning Policies from Data for Zone Temperature Control

Abstract

Replacing poorly performing existing controllers with smarter solutions will decrease the energy intensity of the building sector. Recently, controllers based on Deep Reinforcement Learning (DRL) have been shown to be more effective than conventional baselines. However, since the optimal solution is usually unknown, it is still unclear if DRL agents are attaining near-optimal performance in general or if there is still a large gap to bridge.In this paper, we investigate the performance of DRL agents compared to the theoretically optimal solution. To that end, we leverage Physically Consistent Neural Networks (PCNNs) as simulation environments, for which optimal control inputs are easy to compute. Furthermore, PCNNs solely rely on data to be trained, avoiding the difficult physics-based modeling phase, while retaining physical consistency. Our results hint that DRL agents not only clearly outperform conventional rule-based controllers, they furthermore attain near-optimal performance.

Lessons Learned from Data-Driven Building Control Experiments: Contrasting Gaussian Process-based MPC, Bilevel DeePC, and Deep Reinforcement Learning

“…Most of the recent successes of RL were obtained through NN-based policies, which is then referred to as DRL. A plethora of algorithms to solve such (D)RL optimization problems have been developed in the past few years, and we rely on Twin Delayed Deep Deterministic policy gradients (TD3) [33] for our experiments [27].…”

“…The reward function was designed as r(x, u) = − max{T − b u , 0} − max{b l − T, 0} − λu, where T is the indoor temperature, b u and b l represent the current upper and lower temperature bounds, and λ balances the comfort of the occupants and the energy consumption. More details on the setup and simulation results can be found in [27], where DRL were shown to clearly outperform industrial baselines and attain near-optimal performance.…”

“…In this work, we focus our analysis on three real-world building control experiments using distinct combinations of models and controllers: GP-based MPC [25], bilevel DeePC [26] -relying on behavioral system theory -and DRL matched with NN dynamical models [27]. This selection is representative of the various possible choices as it covers linear, nonlinear, parametric, and non-parametric models, as well as both predictive and explicit control laws.…”