Model-free control of thermostatically controlled loads connected to a district heating network

Abstract

Optimal control of thermostatically controlled loads connected to a district heating network is considered a sequential decisionmaking problem under uncertainty. The practicality of a direct model-based approach is compromised by two challenges, namely scalability due to the large dimensionality of the problem and the system identification required to identify an accurate model. To help in mitigating these problems, this paper leverages on recent developments in reinforcement learning in combination with a market-based multi-agent system to obtain a scalable solution that obtains a significant performance improvement in a practical learning time. The control approach is applied on a scenario comprising 100 thermostatically controlled loads connected to a radial district heating network supplied by a central combined heat and power plant. Both for an energy arbitrage and a peak shaving objective, the control approach requires 60 days to obtain a performance within 65% of a theoretical lower bound on the cost.

Optimal Tracking Strategies for Uncertain Ensembles of Thermostatically Controlled Loads

“…The disadvantage of using such a learning algorithm is that the range space must be discretized and can lead to high computational effort or low accuracy. A summary of the literature survey of TCL models are described in Table 1-1. Literature Description TCL population Control [4] CT, Linear Homogeneous LQR [5] CT, bi-linear Homogeneous non-linear [18] DT, Linear Heterogeneous MPC [19] DT, 2D,Linear Homogeneous with uncertainties MPC [20] CT, probabilistic model Heterogeneous - [22] DT, non-linear Heterogeneous Q-learning As it can be seen from this survey, the controllers used in the past literature for energy management of TCL are MPC or LQR for a linear model and state/output feedback controllers for the non-linear model. Figure 1-4 represents the control achieved by using an MPC for temperature dependent loads.…”

“…The solutionĉ andŵ can be found by minimizing e i.k in a least squares sense. The equation (4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22) does not depend on the system dynamics but only on the state and input measurements. This brings us to the online adaptive Algorithm-5.…”

Demand side management in urban district heating networks

“…Liu et al [18] report significant gains by controlling apartments' SH demand via thermostats in a large-scale pilot test. Claessens et al [19] develop a model-free methodology for DHN control which also reflects the recently fast-growing popularity of employing reinforcement learning techniques in energy management research.…”

Intelligent Electric Water Heater Control with Varying State Information

“…Claessens et al control TCLs in a district heating network and obtain a performance within 65% of a theoreticial lower bound on the cost. However, they propose a handcrafted feature extraction method to evaluate what performance can be obtained from this limited state description [13]. More sophisticated feature extraction techniques are proposed by Claessens et al in [14], where a binning of the state space of a cluster of TCLs allows for a convolutional neural network to be used.…”