Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees

Zhou, Mo; Lu, Jianfeng

doi:10.48550/arxiv.2202.00048

Cited by 3 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Before preceding, the following assumptions are required, which are standard in the theoretical analysis of AC methods (Wu et al 2020;Fu, Yang, and Wang 2020;Yang et al 2019;Zhou and Lu 2022).…”

Section: Main Theorymentioning

confidence: 99%

See 1 more Smart Citation

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

Chen

Duan

Liang

et al. 2023

AAAI

View full text Add to dashboard Cite

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an epsilon-optimal solution with at most an order of epsilon to -2.5 sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

show abstract

Section: Main Theorymentioning

confidence: 99%

“…The zeroth-order methods and the policy iteration method are included for completeness. In particular, we note that Zhou and Lu (2022) analyzed the finite-time convergence under a single-timescale stepsize and multi-sample setting. The analysis requires the strong assumption on the uniform boundedness of the critic parameters.…”

Section: Introductionmentioning

confidence: 99%

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

Chen

Duan

Liang

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

Section: Main Theorymentioning

confidence: 99%

“…Nevertheless, we will present numerical examples to support this assumption. Moreover, the assumption for the existence of stationary distribution is common and has been widely used in Zhou and Lu (2022); Olshevsky and Gharesifard (2022).…”

Section: Main Theorymentioning

confidence: 99%

Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator

Chen¹,

Duan²,

Liang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an -optimal solution with at most an Õ( −2.5 ) sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

show abstract

“…For a general MDP, the best known sample complexity for both single-loop and nested-loop approaches is O(ϵ −2 ) (Chen et al, 2021;Olshevsky & Gharesifard, 2022;Xu et al, 2020a;Suttle et al, 2023). The only exception is (Zhou & Lu, 2022), which is due to the special structure of the LQR problem. These studies all use a fixed function class in the critic, and therefore, the convergence error consists of a non-diminishing constant term of ε critic .…”

Section: Related Workmentioning

confidence: 99%

Service Function Chaining in NFV-Enabled Edge Networks with Natural Actor-Critic Deep Reinforcement Learning

Wang

et al. 2021

2021 IEEE/CIC International Conference on Communications in China (ICCC)

View full text Add to dashboard Cite

Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees

Cited by 3 publications

References 13 publications

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator

Service Function Chaining in NFV-Enabled Edge Networks with Natural Actor-Critic Deep Reinforcement Learning

Contact Info

Product

Resources

About