A Dynamical System Approach to Stochastic Approximations

Abstract

It is known that some problems of almost sure convergence for stochastic approximation processes can be analyzed via an ordinary differential equation (ODE) obtained by suitable averaging. The goal of this paper is to show that the asymptotic behavior of such a process can be related to the asymptotic behavior of the ODE without any particular assumption concerning the dynamics of this ODE. The main results are as follows: a) The limit sets of trajectory solutions to the stochastic approximation recursion are, under classical assumptions, almost surely nonempty compact connected sets invariant under the flow of the ODE and contained in its set of chain-recurrence. b) If the gain parameter goes to zero at a uitable rate depending on the expansion rate of the ODE, any trajectory solution to the recursion is almost surely asymptotic to a forward trajectory solution to the ODE. and Priouret (1990). The main idea of the method is to describe the asymptotic behavior of the algorithm in terms of the behavior of the ODE. For stochastic algorithms having a decreasing gain sequence, the classical result stating the relationship between the algorithm (1) and the ODE (2) has the following form" Let w. be a stable equilibrium for the ODE. If {/}n>_0 goes to zero at a suitable rate and if the sequence {Wn}n>_O enters infinitely often a compact subset of the domain of attraction of w., then {Wn},_>0 converges almost surely toward w . This kind of result has been obtained by Ljung (1977); Kushner and Clark (1978); Mtivier andPriouret (1984, 1987); Benveniste, Mtivier, and Priouret (1990); and Kuan and White (1992), among others, under fairly general conditions. It relies the asymptotic behavior of the algorithm with a strong notion of recurrence for the ODE: the notion of fixed point.With increasing interest in artificial neural networks and due to some limitations of the standard backpropagation algorithm, "heuristic" learning rules for feedforward neural networks have been recently proposed and experimentally studied. The ODE associated with these algorithms is not given by a gradient vectorfield (as is the case for backpropagation), and the classical convergence results on stochastic gradient algorithms cannot be successfully applied. The consideration of these algorithms led us to formulate the following problem:Without any particular assumption on the dynamics of H, is it again possible to describe the asymptotic behavior of (1) in terms of the asymptotic behavior of (2)?The main goal of this paper is to address this question.

Asymptotic bias of stochastic gradient search

“…As the piecewise linear interpolation of sequence {θ n } n≥0 falls into the category of such solutions, the concept of chain-recurrence is tightly connected to the asymptotic behavior of stochastic gradient search. In [5], [6], it has been shown that for unbiased gradient estimates, all limit points of {θ n } n≥0 belong to R and that each element of R can potentially be a limit point of {θ n } n≥0 with a non-zero probability.…”

“…Hence, in general, a limit point of {θ n } n≥0 is in R but not necessarily in S. For more details on chain-recurrence, see [5], [6], [14] and references therein. Given these results, it will prove useful to involve both R and S in the asymptotic analysis of biased stochastic gradient search.…”

A generalized Pólya's urn with graph based interactions

“…In this paper, we study the limiting behavior, as the number of steps grows, of the proportion of balls in the bins of G. More specifically, let N 0 = m i=1 B i (0) denote the initial total number of balls, let 2) be the proportion of balls at vertex i after step n, and let x(n) = (x 1 (n), . .…”

Pooling, pricing and trading of risks

“…6 Motivation for such an 3 The subsequent arguments can accommodate an infinite measure space S together with the Hilbert space Y = L 2 (S, E) of square-integrable profiles s 7 → y s ∈ E, mapping S into a Euclidean space E. 4 That state-contingent claim could quite simply come as a financial credit or debit. Then E = R. Alternatively, if real assets generate various goods, mentioned on a finite list G, then E = R G .…”

“…Since the latter converges, so does the bilateral process by Prop. 2.1 in [3]. ¤ It deserves emphasis that players may very well be scantly informed.…”