Analysis of Stochastic Approximation Schemes With Set-Valued Maps in the Absence of a Stability Guarantee and Their Stabilization

Abstract

In this paper, we analyze the behavior of stochastic approximation schemes with set-valued maps in the absence of a stability guarantee. We prove that after a large number of iterations if the stochastic approximation process enters the domain of attraction of an attracting set it gets locked into the attracting set with high probability. We demonstrate that the above result is an effective instrument for analyzing stochastic approximation schemes in the absence of a stability guarantee, by using it obtain an alternate criteria for convergence in the presence of a locally attracting set for the mean field and by using it to show that a feedback mechanism, which involves resetting the iterates at regular time intervals, stabilizes the scheme when the mean field possesses a globally attracting set, thereby guaranteeing convergence. The results in this paper build on the works of V.S. Borkar, C. Andrieu and H. F. Chen , by allowing for the presence of set-valued drift functions.

Remarks on Differential Inclusion limits of Stochastic Approximation

“…Following the pioneering work of Benaim, Hofbauer and Sorin [4,5], which, among other things, establish a counterpart of Theorem 2.1 for this case, this iteration has been extensively studied in literature, some of it motivated by applications to reinforcement learning [7,13,14,22,23,27,28,29]. The aforementioned extension of Theorem 2.1 is as follows.…”

A Unified Lyapunov Framework for Finite-Sample Analysis of Reinforcement Learning Algorithms

“…shown in [32,33] that the SA algorithm converges almost surely as long as the corresponding ODE is stable. The ODE approach was extended to more general cases in [34,35,36], where the ODE lacks stability, or has multiple equilibrium points. The convergence of various SA algorithms such as SA with Markovian noise and multiple time-scale SA was studied in [37,36] and [38,39] respectively.…”

Stochastic Recursive Inclusions in Two Timescales with Nonadditive Iterate-Dependent Markov Noise

“…Further, adaptive projection-based methods, which are rooted in Chen et al [14] and Chen and Yunmin [15], guarantee stability by truncating iterates when found to be lying outside a prescribed compact set and have been extended to the case with Markov noise by Andrieu et al [1] and Fort et al [18]. Yaji and Bhatnagar [42] extend these adaptive projection schemes to stochastic approximations with set-valued maps without Markov noise.…”

“…and clearly the point H μ (x) −1 ∇J μ (x) is a global attractor of the above o.d.e.. From the result of Kloeden and Kozyakin [22] on continuity of attractors, we have that for any (x, δ), the δ-inflated dynamical system as in Equation ( 43) has a global attractor (denoted by λ δ (x) ⊆ R d 1 ), and for any sequence {(x n , δ n )} converging to (x, δ), H(λ δ n (x n ), λ δ (x)) converges to zero, where H(•) denotes the Hausdorff metric on the family of compact and convex subsets of R d (see equation ( 13) in Yaji and Bhatnagar [42] for a definition). Clearly, for any x, λ 0 (x) H μ (x) −1 ∇J μ (x).…”