Deep Reinforcement Learning for Autonomous Water Heater Control

Abstract

Electric water heaters represent 14% of the electricity consumption in residential buildings. An average household in the United States (U.S.) spends about USD 400–600 (0.45 ¢/L–0.68 ¢/L) on water heating every year. In this context, water heaters are often considered as a valuable asset for Demand Response (DR) and building energy management system (BEMS) applications. To this end, this study proposes a model-free deep reinforcement learning (RL) approach that aims to minimize the electricity cost of a water heater under a time-of-use (TOU) electricity pricing policy by only using standard DR commands. In this approach, a set of RL agents, with different look ahead periods, were trained using the deep Q-networks (DQN) algorithm and their performance was tested on an unseen pair of price and hot water usage profiles. The testing results showed that the RL agents can help save electricity cost in the range of 19% to 35% compared to the baseline operation without causing any discomfort to end users. Additionally, the RL agents outperformed rule-based and model predictive control (MPC)-based controllers and achieved comparable performance to optimization-based control.

Electric Water Boiler Energy Prediction: State-of-the-Art Review of Influencing Factors, Techniques, and Future Directions

“…Autoregression and differencing make the time series stationary [154], while seasonal ARIMA (SARIMA) models account for data seasonality [139]. State-space models and Kalman filter models are useful when the system's underlying states (hot water consumption patterns) are not directly observable but can be estimated from observed data [146]. Machine learning (ML) methods, like recursive neural networks (RNNs) and long short-term memory (LSTM), can detect complex time series patterns, improving prediction accuracy [103,150].…”

“…J. Cao et al [147] DRL Cost and energy minimisation and user comfort. Amasyali et al [146] RL Cost and energy minimisation over time of use (TOU). Xu et al [76] RL Energy minimisation.…”

Electric Water Heater Modeling for Large-Scale Distribution Power Systems Studies with Energy Storage CTA-2045 Based VPP and CVR

“…Successful studies into EWH smart controls include a centrally adapted control model which avoided the peak power by scheduling each EWH, thereby reducing the peak load of 1.05 kW/EWH to 0.4 kW/EWH [17] and a deep Q-networks algorithm for water heaters under T.O.U pricing that showed electricity cost savings up to 35% [18]. Other recent examples of EWH controls to reduce grid impact from electric vehicles (EV) show that thermal energy storage is viable to improve grid operation, given the smart technology to control them is in place [19,20].…”

Reinforcement learning for watershed and aquifer management: a nationwide view in the country of Mexico with emphasis in Baja California Sur

“…Notably, RL has found applications in various water-related domains, including water distribution, heating, water metering, and reservoir operation (Castelletti et al, 2010;Ruelens et al, 2018;Hu et al, 2020Hu et al, , 2022Amasyali et al, 2021;Chen and Ray, 2022;Khampuengson and Wang, 2022). However, integrating rule-based environments within RL for water management simulations is new.…”