Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Cayci, Semih; He, Niao; Srikant, R.

doi:10.48550/arxiv.2106.04096

Cited by 8 publications

(14 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For large state or action space, there has also been fruitful developments on extending policy gradient methods to learning the optimal policy within a parameterized function class [23,17]. The convergence of these methods, on the other hand, seems more subtle than their tabular counterparts, and often requires additional technical assumptions on the function class and the underlying MDP [1,18,25,3].…”

Section: Introductionmentioning

confidence: 99%

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Li¹,

Zhao²,

Lan³

2022

Preprint

View full text Add to dashboard Cite

We propose the homotopic policy mirror descent (HPMD) method for solving discounted, infinite horizon MDPs with finite state and action space, and study its policy convergence. We report three properties that seem to be new in the literature of policy gradient methods: (1) HPMD exhibits global linear convergence of the value optimality gap, and local superlinear convergence of the policy to the set of optimal policies with order γ −2 . The superlinear convergence of the policy takes effect after no more than O(log(1/∆ * )) number of iterations, where ∆ * is defined via a gap quantity associated with the optimal state-action value function; (2) HPMD also exhibits last-iterate convergence of the policy, with the limiting policy corresponding exactly to the optimal policy with the maximal entropy for every state. No regularization is added to the optimization objective and hence the second observation arises solely as an algorithmic property of the HPMD method. (3) For the stochastic HPMD method, we further demonstrate a better than O(|S||A|/ 2 ) sample complexity for small optimality gap , when assuming a generative model for policy evaluation.

show abstract

Section: Introductionmentioning

confidence: 99%

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Li¹,

Zhao²,

Lan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The Fisher-non-degenerate setting implicitly guarantees that the agent is able to explore the state-action space under the considered policy class. Similar conditions of the Fisher-non-degeneracy is also required in other global optimum convergence framework (Assumption 6.5 in [1] on the relative condition number and Assumption 3 in [6] on the regularity of the parametric model). Assumption 13 is satisfied by a wide families of policies, including the Gaussian policy (134) and certain neural policy.…”

Section: A4 Proof Of Rl Resultsmentioning

confidence: 99%

“…We need this assumption because of the specific form of the error of Hessian estimator in recursion inequality in (6). As a result of assumption in (8), using a version of matrix moment inequality (See Lemma 3 in Appendix), we show that the dependency of the Hessian sample complexity in Theorem 1 on dimension d is in the order of log d.…”

Section: Scrn Under Gradient Dominance Property Withmentioning

confidence: 99%

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Masiha¹,

Salehkaleybar²,

He³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property which holds in a wide range of applications in machine learning and signal processing. This condition ensures that any first-order stationary point is a global optimum. We prove that SCRN improves the best-known sample complexity of stochastic gradient descent in achieving -global optimum by a factor of O( −1/2 ). Even under a weak version of gradient dominance property, which is applicable to policy-based reinforcement learning (RL), SCRN achieves the same improvement over stochastic policy gradient methods. Additionally, we show that the sample complexity of SCRN can be improved by a factor of O( −1/2 ) using a variance reduction method with time-varying batch sizes. Experimental results in various RL settings showcase the remarkable performance of SCRN compared to first-order methods. * equal contribution Preprint. Under review.

show abstract

“…Note that, unlike MDPs, there may be only strictly non-deterministic policies for POMDPs [20]. Furthermore, if one employs entropy regularization within the NAC framework, which is commonly employed in practice, π * automatically satisfies Condition 1 [34,35,12].…”

Section: Memory-inference Error Tradeoff For Sliding-block Controllersmentioning

confidence: 99%

Learning to Control Partially Observed Systems with Finite Memory

Cayci¹,

He²,

Srikant³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces, where the controller has access to only noisy observations of the underlying controlled Markov chain. We consider a natural actor-critic method that employs a finite internal memory for policy parameterization, and a multi-step temporal difference learning algorithm for policy evaluation. We establish, to the best of our knowledge, the first non-asymptotic global convergence of actor-critic methods for partially observed systems under function approximation. In particular, in addition to the function approximation and statistical errors that also arise in MDPs, we explicitly characterize the error due to the use of finite-state controllers. This additional error is stated in terms of the total variation distance between the traditional belief state in POMDPs and the posterior distribution of the hidden state when using a finite-state controller. Further, we show that this error can be made small in the case of sliding-block controllers by using larger block sizes.

show abstract

Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Cited by 8 publications

References 13 publications

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Learning to Control Partially Observed Systems with Finite Memory

Contact Info

Product

Resources

About