On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control

Abstract

The optimization landscape of optimal control problems plays an important role in the convergence of many policy gradient methods. Unlike state-feedback Linear Quadratic Regulator (LQR), static output-feedback policies are typically insufficient to achieve good closed-loop control performance. We investigate the optimization landscape of linear quadratic control using dynamic output-feedback policies, denoted as dynamic LQR (dLQR) in this paper. We first show that the dLQR cost varies with similarity transformations. We then derive an explicit form of the optimal similarity transformation for a given observable stabilizing controller. We further characterize the unique observable stationary point of dLQR. This provides an optimality certificate for policy gradient methods under mild assumptions. Finally, we discuss the differences and connections between dLQR and the canonical linear quadratic Gaussian (LQG) control. These results shed light on designing policy gradient algorithms for decisionmaking problems with partially observed information.

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

“…However, beyond these properties, until recently little was known about the geometric and analytical properties of the PO formulation of LQG control. We will mainly summarize results on the optimization landscape of LQG control from Zheng et al (63), especially with respect to the connectivity of the stabilizing set K and the structure of the stationary points; several related extensions can be found in other works (64)(65)(66)(67). Before introducing the results, we discuss a special structure for LQG control with the state-space dynamical controller parameterization in Equation 11.…”

“…Recent theoretical results on PO for particular classes of control synthesis problems, some of which are discussed in this survey, not only are exciting but also lead to a new research thrust at the interface of control theory and machine learning. This survey includes control synthesis related to linear quadratic regulator (LQR) theory (35)(36)(37)(38)(39)(40)(41)(42)(43)(44), stabilization (45)(46)(47), linear robust/risk-sensitive control (48)(49)(50)(51)(52)(53)(54)(55), Markov jump linear quadratic control (56-59), Lur'e system control (60), output feedback control (61)(62)(63)(64)(65)(66)(67), and dynamic filtering (68). Surprisingly, some of these strong global convergence results for PO have been obtained in the absence of convexity in the design objective and/or the underlying feasible set.…”

Connectivity of the Feasible and Sublevel Sets of Dynamic Output Feedback Control With Robustness Constraints

“…This brings some positive news and opens the possibility of developing global convergent policy search methods for dynamical output feedback problems, such as linear quadratic Gaussian (LQG) control [16]. Two other recent studies are [17], [18]. In [18], the global convergence of policy search over dynamical filters was proved for a simpler estimation problem.…”

On the Sample Complexity of Stabilizing Linear Systems via Policy Gradient Methods

“…The seminal work of [8] first shows that PG methods have global convergence guarantees for the celebrated linear quadratic regulator (LQR) problem. Then, many sample complexity results of PG methods are established for both discrete-time [10], [11] and continuoustime LQR [12], and the PG methods are applied to solve other fundamental Linear Quadratic (LQ) problems, such as risk-sensitive control [13], LQ game [14], linear quadratic Gaussian (LQG) [15], [16] and decentralized control [17], [18], just to name a few. Though these advances lead to fruitful and profound results for model-free control synthesis, they all require a common assumption: a stabilizing controller must be known a prior.…”