Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedback

Duan, Jingliang; Li, Jie; Zhao, Lin

doi:10.48550/arxiv.2109.13132

Cited by 4 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent theoretical results on PO for particular classes of control synthesis problems, some of which are discussed in this survey, not only are exciting but also lead to a new research thrust at the interface of control theory and machine learning. This survey includes control synthesis related to linear quadratic regulator (LQR) theory (35)(36)(37)(38)(39)(40)(41)(42)(43)(44), stabilization (45)(46)(47), linear robust/risk-sensitive control (48)(49)(50)(51)(52)(53)(54)(55), Markov jump linear quadratic control (56-59), Lur'e system control (60), output feedback control (61)(62)(63)(64)(65)(66)(67), and dynamic filtering (68). Surprisingly, some of these strong global convergence results for PO have been obtained in the absence of convexity in the design objective and/or the underlying feasible set.…”

Section: Introductionmentioning

confidence: 99%

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Zhang

et al. 2023

Annu. Rev. Control Robot. Auton. Syst.

View full text Add to dashboard Cite

Gradient-based methods have been widely used for system design and optimization in diverse application domains. Recently, there has been a renewed interest in studying theoretical properties of these methods in the context of control and reinforcement learning. This article surveys some of the recent developments on policy optimization, a gradient-based iterative approach for feedback control synthesis that has been popularized by successes of reinforcement learning. We take an interdisciplinary perspective in our exposition that connects control theory, reinforcement learning, and large-scale optimization. We review a number of recently developed theoretical results on the optimization landscape, global convergence, and sample complexityof gradient-based methods for various continuous control problems, such as the linear quadratic regulator (LQR), [Formula: see text] control, risk-sensitive control, linear quadratic Gaussian (LQG) control, and output feedback synthesis. In conjunction with these optimization results, we also discuss how direct policy optimization handles stability and robustness concerns in learning-based control, two main desiderata in control engineering. We conclude the survey by pointing out several challenges and opportunities at the intersection of learning and control.

show abstract

Section: Introductionmentioning

confidence: 99%

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Zhang

et al. 2023

Annu. Rev. Control Robot. Auton. Syst.

View full text Add to dashboard Cite

show abstract

“…This is also known as Partially Observable Markov Decision Process (POMDP) in the Markovian system setting [18]. Some recent works have studied static output-feedback (SOF) controllers to optimize a linear quadratic cost function [10], [19]- [21]. Unlike state-feedback LQR problems, it is shown that policy gradient methods are unlikely to find the globally optimal SOF controller.…”

Section: Introductionmentioning

confidence: 99%

“…This paper takes a step further to analyze the optimization landscape of Dynamic output-feedback LQR (dLQR). Unlike the vanilla LQR and the SOF that use static feedback policies [10], [19]- [21], the problem of dLQR searches over the set of dynamic controllers, which has rich yet complicated landscape properties. The recent work [25], [26] has analyzed the structure of optimal dynamic controllers for the classical Linear Quadratic Gaussian (LQG) control problem.…”

Section: Introductionmentioning

confidence: 99%

On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control

Duan¹,

Cao²,

Zheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The optimization landscape of optimal control problems plays an important role in the convergence of many policy gradient methods. Unlike state-feedback Linear Quadratic Regulator (LQR), static output-feedback policies are typically insufficient to achieve good closed-loop control performance. We investigate the optimization landscape of linear quadratic control using dynamic output-feedback policies, denoted as dynamic LQR (dLQR) in this paper. We first show that the dLQR cost varies with similarity transformations. We then derive an explicit form of the optimal similarity transformation for a given observable stabilizing controller. We further characterize the unique observable stationary point of dLQR. This provides an optimality certificate for policy gradient methods under mild assumptions. Finally, we discuss the differences and connections between dLQR and the canonical linear quadratic Gaussian (LQG) control. These results shed light on designing policy gradient algorithms for decisionmaking problems with partially observed information.

show abstract

“…To this end, we analyze the classic single-sample AC for solving the Linear Quadratic Regulator (LQR), a fundamental control task which is commonly employed as a testbed to explore the behavior and limits of RL algorithms under continuous state-action spaces (Fazel et al 2018;Yang et al 2019;Tu and Recht 2018;Krauth, Tu, and Recht 2019;Duan, Li, and Zhao 2021). In the LQR case, the Q-function is a linear function of the quadratic form of state and action.…”

Section: Introductionmentioning

confidence: 99%

Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator

Chen¹,

Duan²,

Liang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an -optimal solution with at most an Õ( −2.5 ) sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

show abstract

Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedback

Cited by 4 publications

References 22 publications

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control

Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator

Contact Info

Product

Resources

About