Kullback-Leibler-Quadratic Optimal Control of Flexible Power Demand

Abstract

This paper presents advances in Kullback-Leibler-Quadratic (KLQ) optimal control: a stochastic control framework for Markovian models. The motivation is distributed control of large networks. As in prior work, the objective function is composed of a state cost in the form of Kullback-Leibler divergence plus a quadratic control cost. With this choice of objective function, the optimal probability distribution of a population of agents over a finite time horizon is shown to be an exponential tilting of the nominal probability distribution. The same is true for the controlled transition matrices that induce the optimal probability distribution. However, one limitation of the previous work is that randomness can only be introduced via the control policy; all uncontrolled (natural) processes must be modeled as deterministic to render them immutable under an exponential tilting. In this work, only the controlled dynamics are subject to tilting, allowing for more general probabilistic models.Another advancement is a reduction in complexity based on lossy compression using transform techniques. This is motivated by the need to consider time horizons that are much longer than the inter-sampling times required for reliable control. Numerical experiments are performed in a power network setting. The results show that the KLQ method enables the aggregate power consumption of a collection of flexible loads to track a time-varying reference signal, while simultaneously ensuring each individual load satisfies its own quality of service constraints.

Kullback-Leibler-Quadratic Optimal Control in a Stochastic Environment

“…Satisfying the energy requirements of this signal would require refrigerators to violate their temperature bounds. This is not possible since quality of service is guaranteed with our distributed control control architecture, where switching decisions are still made at the load level [1]. Notice how the collective power consumption is gracefully truncated at the peak and valley of the reference signal while tracking remains nearly perfect at all other times.…”

“…The stochastic process {∆ k (X k−1 , S k )} is a martingale difference sequence; it vanishes when nature is deterministic, reducing to the solution obtained in [1], [2].…”

“…We have found in examples that using gradient ascent on the dual function curve may be slow to converge, likely due to a large "overshoot" when applying standard first-order methods [1]. In the numerical results that follow we opt for proximal gradient methods [26].…”

Probabilistic design of optimal sequential decision-making algorithms in learning and control

“…We close this paragraph by recalling [126,127,128] where closely related infinite-dimensional finite-horizon control problems are considered in the context of distributed control and energy systems. In such papers, control formulations are considered where state transition probabilities can be shaped directly together with the presence of an additional (quadratic) cost criterion.…”

Optimal Decision-Making for Autonomous Agents via Data Composition

“…divergence minimization and we refer to e.g., [15], [16] for examples across learning and control that involve minimizing this functional. Further, the study of mechanisms enabling agents to re-use data, also arises in the design of prediction algorithms from experts [17] and of learning algorithms from multiple simulators [18].…”