Intelligent Electric Water Heater Control with Varying State Information

Abstract

The increasing share of renewable energy sources in the electricity grid results in a higher degree of uncertainty regarding electrical energy production. In response to this, flexibility of the demand has been proposed as part of the solution. An important source of flexibility available at the residential consumer side are thermostatically controlled loads (TCLs). In this paper the activation of this source of flexibility is achieved by applying batch reinforcement learning (BRL) to an electric water heater (EWH) in a Time of Use (ToU) setting. The cost performance of six BRL agents with six different state spaces is compared quantitatively. In every case, the BRL agent can successfully shift energy consumption within 20-25 days. The performance of an agent with access to multiple temperature sensors along the height of the EWH is comparable to the performance of an agent with access to only the highest temperature sensor. This indicates manufacturing costs related to sensors can be reduced while maintaining the same performance. Additionally, results show that the inclusion of a theoretical state of charge value in the state space increases performance by more than 8% compared to the performance of the other BRL agents. It is therefore argued that an estimation of the state of charge should be included in future work as it would increase cost performance.

Domain Randomization for Demand Response of an Electric Water Heater

“…The stratified buffer model we use has been presented and validated in the lab by Vanthournout et al [14]. This model has been used in DR research several times [1,16,17], and for more details we refer to Vanthournout et al [14].…”

“…Demand response of an EWH gives rise to a Sequential Decision Problem (SDP). In earlier work [1,3,16,17] the learning agent's SDP has already been formulated as a Partially Observable Markov Decision Process (POMDP). This section reviews the POMDP and introduces the algorithms used in our work.…”

“…1) State-space: Patyn et al [16] investigated the influence of different observable state-space configurations on the performance of a RL agent applied to an EWH. Observing SoC results in the best performing control policy.…”

“…Action u = 1 implies the EWH's heating element is turned on at rated power, u = 0 turns the heating element off. However, the EWH's internal controller maps the agent's action u to a physical action u phys according to (16), with T b the temperature at the backup sensor location. (16) While H(u) is unknown to the agent, it's effect can be observed through the cost-function.…”

“…Due to its proven track record in DR applications we also consider FQI [1,2,16] in our experiments. In a discrete-time MDP with a one day horizon (T = 96) we can reformulate the SDP as a sequence of T control problems.…”