Concentration bounds for two time scale stochastic approximation

Borkar, Vivek S.; Pattathil, Sarath

doi:10.1109/allerton.2018.8636078

Cited by 19 publications

(19 citation statements)

References 13 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Actor-critic method was first proposed in [20] as a two-time-scale stochastic approximation [5,6,9,19] variant of the policy gradient algorithm [37], where a faster time scale is used to collect samples for gradient estimation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithm

Khodadadian¹,

Doan²,

Romberg³

et al. 2021

Preprint

View full text Add to dashboard Cite

Actor-critic style two-time-scale algorithms are very popular in reinforcement learning, and have seen great empirical success. However, their performance is not completely understood theoretically. In this paper, we characterize the global convergence of an online natural actor-critic algorithm in the tabular setting using a single trajectory. Our analysis applies to very general settings, as we only assume that the underlying Markov chain is ergodic under all policies (the so-called Recurrence assumption). We employ -greedy sampling in order to ensure enough exploration.For a fixed exploration parameter , we show that the natural actor critic algorithm is O( 1 T 1/4 + ) close to the global optimum after T iterations of the algorithm.By carefully diminishing the exploration parameter as the iterations proceed, we also show convergence to the global optimum at a rate of O(1/T 1/6 ).

show abstract

Section: Related Workmentioning

confidence: 99%

“…The term ξ t /t is due to the variation of t over time, and the term β t is due to the exponential weight update of policy π t in Algorithm 1. Hence, by definition of θ t in (6) and by the update (9), and due to the Lipschitzness of Q π function, we have…”

Section: T T +1mentioning

confidence: 99%

Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithm

Khodadadian¹,

Doan²,

Romberg³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Notable examples are [7,8] which prove asymptotic convergence of TD(λ). Recently, finite-time performance of single-agent stochastic approximation and TD algorithms has been studied in [9][10][11][12][13][14][15][16][17]; many other works have now appeared that perform finitetime analysis for other RL algorithms, see, e.g., [18][19][20][21][22][23][24][25][26][27][28], just to name a few. Many distributed reinforcement learning algorithms have now been proposed in the literature.…”

Section: Related Workmentioning

confidence: 99%

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

Ye¹,

Gupta²,

Liu³

2021

Preprint

View full text Add to dashboard Cite

This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm.

show abstract

“…We end by pointing at some recent papers that build upon the ideas discussed here, thereby illustrating the usefulness of this work. In [14] and [8], concentration bounds have been obtained for two-timescale SA; the first one deals with the linear case, while the second one handles the generic nonlinear setup. Separately, [24] studies constant stepsize SA used to track a slowly moving target and provides bounds on the tracking error.…”

Section: Sincementioning

confidence: 99%

A Concentration Bound for Stochastic Approximation via Alekseev’s Formula

Thoppe

Borkar

2019

Stochastic Systems

View full text Add to dashboard Cite

Given an ODE and its perturbation, the Alekseev formula expresses the solutions of the latter in terms related to the former. By exploiting this formula and a new concentration inequality for martingale-differences, we develop a novel approach for analyzing nonlinear Stochastic Approximation (SA). This approach is useful for studying a SA's behaviour close to a Locally Asymptotically Stable Equilibrium (LASE) of its limiting ODE; this LASE need not be the limiting ODE's only attractor. As an application, we obtain a new concentration bound for nonlinear SA. That is, given ǫ > 0 and that the current iterate is in a neighbourhood of a LASE, we provide an estimate for i.) the time required to hit the ǫ−ball of this LASE, and ii.) the probability that after this time the iterates are indeed within this ǫ−ball and stay there thereafter. The latter estimate can also be viewed as the 'lock-in' probability. Compared to related results, our concentration bound is tighter and holds under significantly weaker assumptions. In particular, our bound applies even when the stepsizes are not square-summable. Despite the weaker hypothesis, we show that the celebrated Kushner-Clark lemma continues to hold. * Supported partially by an IBM PhD fellowship. † Supported in part by a J. C. Bose Fellowship and a grant 'Approximation of High Dimensional Optimization and Control Problems' from the Department of Science and Technology, Government of India.MSC 2010 subject classifications: Primary 62L20; secondary 93E25, 60G42, 34D10

show abstract

Concentration bounds for two time scale stochastic approximation

Cited by 19 publications

References 13 publications

Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithm

Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithm

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

A Concentration Bound for Stochastic Approximation via Alekseev’s Formula

Contact Info

Product

Resources

About