Comparing neural architectures for demand response through model-free reinforcement learning for heat pump control

Abstract

As batch reinforcement learning algorithms reach maturity and neural networks are used increasingly in reinforcement learning, a performance comparison of these models should be performed. This paper discusses the implementation of a heat pump agent in a demand response setting and its cost effectiveness when implemented with different neural network types. The agent maintains the interior air temperature of a building between pre-set temperature constraints, with four actions at its disposal. The agent is incentivized to shift loads in a day-ahead market in order to minimize daily electricity costs. The simulation considered a multilayer perceptron (MLP), a convolutional neural network (CNN) and a long short-term memory neural network (LSTM) to model the environment dynamics. All architectures outperform a trivial thermostat controller and shift loads successfully after 20-25 days. For this particular setup, there is no significant difference between the MLP and the LSTM, while they do outperform the CNN model. The MLP is preferred as it requires far less computation time.

Domain Randomization for Demand Response of an Electric Water Heater

“…In a first application, Ruelens et al [1] illustrate that a RL method called Fitted Q-Iteration (FQI) reduces the electricity consumption cost of an Electric Water Heater (EWH) by 24 %, when charged with Belgian day-ahead electricity prices. Similar results have been obtained for spaceheating [3,4]. In a second application, De Somer et al [5] use EWH storage and FQI for local photovoltaic (PV) selfconsumption.…”

A review of reinforcement learning for autonomous building energy management

“…The authors report energy savings of 13 and 23% and lowered discomfort ratings by 62 and 80% in the two buildings tested. In a 2018 study, Patyn et al compared the performance difference of Q learning on three different neural network architectures (convolutional neural networks (CNN), LSTM networks and a feed forward multi-layer neural network) for HVAC control [73]. Their results indicate that the LSTM and multi-layer network perform best however the LSTM training time was twice as long.…”

Demand Response Management for Industrial Facilities: A Deep Reinforcement Learning Approach

“…In [26], De Somer et al demonstrated a data-driven control approach for DR in residential buildings, using RL to optimally schedule the heating cycles of domestic hot water buffers to maximize the self-consumption of local photovoltaic production. In [27], Patyn et al discussed the implementation of RL for heat pumps in a DR setting and its cost-effectiveness in comparison to different types of neural networks. The increasing utilization of EVs holds a great DR potential by their electrical storage capacity as well as their inherent connectivity [21].…”