A Tale of Two-Timescale Reinforcement Learning with the Tightest Finite-Time Bound

Dalal, Gal; Szörényi, Balázs; Thoppe, Gugan

doi:10.1609/aaai.v34i04.5779

Cited by 31 publications

(32 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, it would be interesting to extend weak convergence rate analysis of (Konda and Tsitsiklis 2004;Mokkadem and Pelletier 2006) to Three-TS regime. Also, extending the recent convergence rate results in expectation and high probability of GTD methods (Dalal et al 2018b;Gupta, Srikant, and Ying 2019;Kaledin et al 2019;Dalal, Szorenyi, and Thoppe 2020) to these momentum settings would be interesting works for the future.…”

Section: Related Work and Conclusionmentioning

confidence: 85%

Gradient Temporal Difference with Momentum: Stability and Convergence

Deb¹,

Bhatnagar²

2021

Preprint

View full text Add to dashboard Cite

Gradient temporal difference (Gradient TD) algorithms are a popular class of stochastic approximation (SA) algorithms used for policy evaluation in reinforcement learning. Here, we consider Gradient TD algorithms with an additional heavy ball momentum term and provide choice of step size and momentum parameter that ensures almost sure convergence of these algorithms asymptotically. In doing so, we decompose the heavy ball Gradient TD iterates into three separate iterates with different step sizes. We first analyze these iterates under one-timescale SA setting using results from current literature. However, the one-timescale case is restrictive and a more general analysis can be provided by looking at a three-timescale decomposition of the iterates. In the process we provide the first conditions for stability and convergence of general threetimescale SA. We then prove that the heavy ball Gradient TD algorithm is convergent using our three-timescale SA analysis. Finally, we evaluate these algorithms on standard RL problems and report improvement in performance over the vanilla algorithms.

show abstract

Section: Related Work and Conclusionmentioning

confidence: 85%

Gradient Temporal Difference with Momentum: Stability and Convergence

Deb¹,

Bhatnagar²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Another intriguing future direction is the setting with dynamic communication protocols, wherein the gossip matrix also evolves with time Doan et al [2019Doan et al [ , 2021. A third direction is that of twotimescale DSA schemes Dalal et al [2018bDalal et al [ , 2020. On the MARL side, important algorithms like distributed Q-learning Lauer and Riedmiller [2000] and its variants need more careful analysis and we believe our techniques would be instrumental for this as well.…”

Section: Discussionmentioning

confidence: 99%

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Thoppe¹,

Kumar²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In Multi-Agent Reinforcement Learning (MARL), multiple agents interact with a common environment, as also with each other, for solving a shared problem in sequential decision-making. It has wide-ranging applications in gaming, robotics, finance, etc. In this work, we derive a novel law of iterated logarithm for a family of distributed nonlinear stochastic approximation schemes that is useful in MARL. In particular, our result describes the convergence rate on almost every sample path where the algorithm converges. This result is the first of its kind in the distributed setup and provides deeper insights than the existing ones, which only discuss convergence rates in the expected or the CLT sense. Importantly, our result holds under significantly weaker assumptions: neither the gossip matrix needs to be doubly stochastic nor the stepsizes square summable. As an application, we show that, for the stepsize n −γ with γ ∈ (0, 1), the distributed TD(0) algorithm with linear function approximation has a convergence rate of O( √ n −γ ln n) a.s.; for the 1/n type stepsize, the same is O( √ n −1 ln ln n) a.s. These decay rates do not depend on the graph depicting the interactions among the different agents.Preprint. Under review.

show abstract

“…The convergence of the two-time-scale SA is traditionally established through the analysis of an associated ordinary differential equation (ODE) [2]. Finite-time convergence of the two-time-scale SA has been studied for both linear [4,5,7,11,12,14,16,19] (when F and G are linear) and nonlinear settings [6,8,24], under both i.i.d. and Markovian samples.…”

Section: Related Workmentioning

confidence: 99%

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Zeng¹,

Doan²,

Romberg³

2021

Preprint

View full text Add to dashboard Cite

We study a novel two-time-scale stochastic gradient method for solving optimization problems where the gradient samples are generated from a time-varying Markov random process parameterized by the underlying optimization variable. These time-varying samples make the stochastic gradient biased and dependent, which can potentially lead to the divergence of the iterates. To address this issue, we consider a two-time-scale update scheme, where one scale is used to estimate the true gradient from the Markovian samples and the other scale is used to update the decision variable with the estimated gradient. While these two iterates are implemented simultaneously, the former is updated "faster" (using bigger step sizes) than the latter (using smaller step sizes). Our first contribution is to characterize the finite-time complexity of the proposed two-time-scale stochastic gradient method. In particular, we provide explicit formulas for the convergence rates of this method under different objective functions, namely, strong convexity, convexity, non-convexity under the PŁ condition, and general non-convexity.Our second contribution is to apply our framework to study the performance of the popular actor-critic methods in solving stochastic control and reinforcement learning problems. First, we study an online natural actor-critic algorithm for the linear-quadratic regulator and show that a convergence rate of Opk ´2{3 q is achieved. This is the first time such a result is known in the literature. Second, we look at the standard online actor-critic algorithm over finite state and action spaces and derive a convergence rate of Opk ´2{5 q, which recovers the best known rate derived specifically for this problem. Finally, we support our theoretical analysis with numerical simulations where the convergence rate is visualized.

show abstract

A Tale of Two-Timescale Reinforcement Learning with the Tightest Finite-Time Bound

Cited by 31 publications

References 9 publications

Gradient Temporal Difference with Momentum: Stability and Convergence

Gradient Temporal Difference with Momentum: Stability and Convergence

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

Contact Info

Product

Resources

About