Ensemble Control of Cycling Energy Loads: Markov Decision Approach

Abstract

A Markov decision process (MDP) framework is adopted to represent ensemble control of devices with cyclic energy consumption patterns, e.g., thermostatically controlled loads. Specifically we utilize and develop the class of MDP models previously coined linearly solvable MDPs, that describe optimal dynamics of the probability distribution of an ensemble of many cycling devices. Two principally different settings are discussed. First, we consider optimal strategy of the ensemble aggregator balancing between minimization of the cost of operations and minimization of the ensemble welfare penalty, where the latter is represented as a KL-divergence between actual and normal probability distributions of the ensemble. Then, second, we shift to the demand response setting modeling the aggregator's task to minimize the welfare penalty under the condition that the aggregated consumption matches the targeted time-varying consumption requested by the system operator. We discuss a modification of both settings aimed at encouraging or constraining the transitions between different states. The dynamic programming feature of the resulting modified MDPs is always preserved; however, 'linear solvability' is lost fully or partially, depending on the type of modification. We also conducted some (limited in scope) numerical experimentation using the formulations of the first setting. We conclude by discussing future generalizations and applications.

Distributed Control Design for Balancing the Grid Using Flexible Loads

“…These ideas were first applied to demand-dispatch in [35], and have seen many extensions since. For more history the reader is referred to [36,14], in addition to the papers surveyed in Section 3.2. While beyond the scope of this article, it is important to note that Todorov's 'linearly solvable' MDP model [44] is similar to prior work such as [24], and the form of the solution could have been anticipated from well-known results in the theory of large-deviations for Markov chains [7].…”

“…The formulation of the IPD optimization problem is unchanged, but its solution is not in the form (14). A computational ODE approach is introduced in [7,6]: for a vector field V whose domain and range are functions on X × X u ,…”

Power of Ensemble Diversity and Randomization for Energy Aggregation

“…Moreover, following the logic of [16,21], it is practically appropriate to also consider the resulting Markov Process (MP) model in discrete time. In fact, this MP formulation is also practically advantageous for analysis of the aforementioned optimal control, where the problem becomes of the Markov decision process (MDP) type, as in [3,4,8]. We would argue that the MP and MDP approaches are naturally appropriate and algorithmically attractive to account for the randomization effects analyzed in the manuscript.…”

“…Time evolution of the energy consumption to its equilibrium value |N ↑ (t) − 1/2| is shown. Four subfigures correspond to the four different DPD cases: Gaussian (top left), Laplacian (top right), Lorentzian (bottom left), and uniform finite-support (bottom right), described by Eqs (7,9,8,10),. respectively.…”

Optimal Load Ensemble Control in Chance-Constrained Optimal Power Flow

“…In [5]- [9], each TCL ensemble is modeled as a discrete-time, discrete-space MDP that is well suited for capturing stochastic dynamics of individual TCLs and is computationally scalable to accommodate hundreds of TCLs in each ensemble. The models [7]- [9] exploit naive economic dispatch frameworks that co-optimize the flexibility of TCL ensembles and distribution system operations. The common caveat of [7]- [9] is that network constraints are neglected and, as a result, these models do not ensure compliance with power flow and voltage limits.…”