A Concentration Bound for Stochastic Approximation via Alekseev’s Formula

Abstract

Given an ODE and its perturbation, the Alekseev formula expresses the solutions of the latter in terms related to the former. By exploiting this formula and a new concentration inequality for martingale-differences, we develop a novel approach for analyzing nonlinear Stochastic Approximation (SA). This approach is useful for studying a SA's behaviour close to a Locally Asymptotically Stable Equilibrium (LASE) of its limiting ODE; this LASE need not be the limiting ODE's only attractor. As an application, we obtain a new concentration bound for nonlinear SA. That is, given ǫ > 0 and that the current iterate is in a neighbourhood of a LASE, we provide an estimate for i.) the time required to hit the ǫ−ball of this LASE, and ii.) the probability that after this time the iterates are indeed within this ǫ−ball and stay there thereafter. The latter estimate can also be viewed as the 'lock-in' probability. Compared to related results, our concentration bound is tighter and holds under significantly weaker assumptions. In particular, our bound applies even when the stepsizes are not square-summable. Despite the weaker hypothesis, we show that the celebrated Kushner-Clark lemma continues to hold. * Supported partially by an IBM PhD fellowship. † Supported in part by a J. C. Bose Fellowship and a grant 'Approximation of High Dimensional Optimization and Control Problems' from the Department of Science and Technology, Government of India.MSC 2010 subject classifications: Primary 62L20; secondary 93E25, 60G42, 34D10

Concentration bounds for two time scale stochastic approximation

“…with Φ y (s, s, y 0 ) = I,D being the Jacobian matrix of g(λ(·), ·). As shown in Lemma 5.3, [15], there exists K, κ y > 0 so that the following holds for t ≥ s:…”

“…Here y(t) ≡ y, x(t) ≡ λ(y) is a constant trajectory and Φ x (·) satisfies the linear system: (12) with initial condition Φ x (t, s, x 0 , y 0 ) = I, where D is the Jacobian matrix of h(·, y). As shown in Lemma 5.3, [15], there exist K, κ x > 0 so that the following holds for t ≥ s and x 0 ∈ V r :…”

“…S. Pattathil is with the Department of Electrical Engineering, IIT Bombay, Powai, Mumbai 400076, India sarathpattathil@iitb.ac.in 1 As in [15], we can relax the second of these conditions to an, bn → 0. We do not discuss this here.…”

“…The aim of the present work is to provide a concentration result for the two time scale scheme in the spirit of [15], which does so for the single time scale case. This may be viewed as a step towards the aforementioned objective.…”

“…We also assume A4 from [15]: There exist a Lyapunov function V ∈ C 1 (R s ) with lim y ↑∞ V (y) = ∞, ∇V (y), g(λ(y), y) < 0 for y = y * , and r > r 0 > 0, ǫ 0 > 0 such that for ǫ < ǫ 0 ,…”