These notes were written for a D.E.A course given at Ecole Normale Supdrieure de Cachan during the 1996-97 and 1997-98 academic years and at University Toulouse III during the 1997-98 academic year. Their aim is to introduce the reader to the dynamical system aspects of the theory of stochastic approximations.

We apply the theoretical results on "stochastic approximations and differential inclusions" developed in Benaïm, Hofbauer and Sorin (2005) to several adaptive processes used in game theory including: classical and generalized approachability, no-regret potential procedures (Hart and Mas-Colell), smooth fictitious play (Fudenberg and Levine).

This paper provides deterministic approximation results for stochastic processes that arise when finite populations recurrently play finite games. The processes are Markov chains, and the approximation is defined in continuous time as a system of ordinary differential equations of the type studied in evolutionary game theory. We establish precise connections between the long-run behavior of the discrete stochastic process, for large populations, and its deterministic flow approximation. In particular, we provide probabilistic bounds on exit times from and visitation rates to neighborhoods of attractors to the deterministic flow. We sharpen these results in the special case of ergodic processes.

It is known that some problems of almost sure convergence for stochastic approximation processes can be analyzed via an ordinary differential equation (ODE) obtained by suitable averaging. The goal of this paper is to show that the asymptotic behavior of such a process can be related to the asymptotic behavior of the ODE without any particular assumption concerning the dynamics of this ODE. The main results are as follows: a) The limit sets of trajectory solutions to the stochastic approximation recursion are, under classical assumptions, almost surely nonempty compact connected sets invariant under the flow of the ODE and contained in its set of chain-recurrence. b) If the gain parameter goes to zero at a uitable rate depending on the expansion rate of the ODE, any trajectory solution to the recursion is almost surely asymptotic to a forward trajectory solution to the ODE. and Priouret (1990). The main idea of the method is to describe the asymptotic behavior of the algorithm in terms of the behavior of the ODE. For stochastic algorithms having a decreasing gain sequence, the classical result stating the relationship between the algorithm (1) and the ODE (2) has the following form" Let w. be a stable equilibrium for the ODE. If {/}n>_0 goes to zero at a suitable rate and if the sequence {Wn}n>_O enters infinitely often a compact subset of the domain of attraction of w., then {Wn},_>0 converges almost surely toward w . This kind of result has been obtained by Ljung (1977); Kushner and Clark (1978); Mtivier andPriouret (1984, 1987); Benveniste, Mtivier, and Priouret (1990); and Kuan and White (1992), among others, under fairly general conditions. It relies the asymptotic behavior of the algorithm with a strong notion of recurrence for the ODE: the notion of fixed point.With increasing interest in artificial neural networks and due to some limitations of the standard backpropagation algorithm, "heuristic" learning rules for feedforward neural networks have been recently proposed and experimentally studied. The ODE associated with these algorithms is not given by a gradient vectorfield (as is the case for backpropagation), and the classical convergence results on stochastic gradient algorithms cannot be successfully applied. The consideration of these algorithms led us to formulate the following problem:Without any particular assumption on the dynamics of H, is it again possible to describe the asymptotic behavior of (1) in terms of the asymptotic behavior of (2)?The main goal of this paper is to address this question.