Dynamics of stochastic approximation algorithms

Benaı̈m, Michel

doi:10.1007/bfb0096509

Cited by 315 publications

(716 citation statements)

References 33 publications

Supporting

Mentioning

709

Contrasting

Unclassified

Order By: Relevance

“…This is exactly what is done in Laslier et al (2001), Lemma 1, where, using terminology and results from Benaim (1999), it is shown that the replicator dynamics is an asymptotic-pseudo-trajectory of the learning process. Results of this type are useful in understanding the asymptotics of the reinforcement learning process, since they allow to show that Theorem 7.3 in Benaim (1999) applies and that the probability that the reinforcement learning process gets absorbed in an asymptotically stable Nash Equilibrium is strictly positive. However, to address general properties of convergence to Nash equilibria of reinforcement learning models, one needs to rule out convergence to all the other rest points of the replicator dynamics.…”

Section: An Overviewsupporting

confidence: 55%

“…0 for t ! 1 4 In essence, the reason why this approximation result holds for random step sizes is related to that stated in Remark 4.3 of Benaim (1999) and therein references, namely that the approximation results of this type hold also for stochastic step sizes, when these are measurable with respect to =fx(n)g and square summable. This could o¤er an alternative way to prove Lemma 1.…”

Section: Notesmentioning

confidence: 76%

“…Although conceptually this would amount to considering di¤erent learning models, the results of Benaim (1999) on shadowing do support this conjecture.…”

Section: Discussionmentioning

confidence: 93%

“…The results we obtain rely on stochastic approximation techniques (Ljung (1978), Arthur et al (1987), (1988), Benaim (1999)) to establish the close connection between the reinforcement learning process and the underlying deterministic replicator equation. Speci…cally the paper shows that up to an error term the behaviour of the stochastic process is well described by a system of discrete time di¤erence equation of the replicator type (Lemma 2).…”

Section: Introductionmentioning

confidence: 79%

“…Theorem 1 of Ljung (1978) or Corollary 6.6 of Benaim (1999) provide convergence conditions, that do straightforwardly apply to our setting whenever a Ljapunov function can be identi…ed. Hence, convergence of the reinforcement learning process obtains for wide classes of underlying games (see Hofbauer and Sigmund (1988) and Weibull (1995) among others on the study of Ljapunov convergence for some classes of games).…”

Section: An Overviewmentioning

confidence: 99%

See 4 more Smart Citations

Learning strict Nash equilibria through reinforcement

Ianni

2014

Journal of Mathematical Economics

View full text Add to dashboard Cite

This paper studies the analytical properties of the reinforcement learning model proposed in Erev and Roth (1998), also termed cumulative reinforcement learning in Laslier et al. (2001). The stochastic model of learning accounts for two main elements: the Law of E¤ect (positive reinforcement of actions that perform well) and the Power Law of Practice (learning curves tend to be steeper initially).The paper establishes a relation between the learning process and the underlying deterministic replicator equation. The main results show that if the solution trajectories of the latter converge su¢ ciently fast, then the probability that all the realizations of the learning process over a given spell of time, possibly in…nite, becomes arbitrarily close to one, from some time on. In particular, the paper shows that the property of fast convergence is always satis…ed in proximity of a strict Nash equilibrium.The results also provide an explicit estimate of the approximation error that could prove to be useful in empirical analysis.JEL: C72, C92, D83.

show abstract

Section: An Overviewsupporting

confidence: 55%

Section: Notesmentioning

confidence: 76%

“…Although conceptually this would amount to considering di¤erent learning models, the results of Benaim (1999) on shadowing do support this conjecture.…”

Section: Discussionmentioning

confidence: 93%

Section: Introductionmentioning

confidence: 79%

Section: An Overviewmentioning

confidence: 99%

See 3 more Smart Citations

Learning strict Nash equilibria through reinforcement

Ianni

2014

Journal of Mathematical Economics

View full text Add to dashboard Cite

show abstract

High‐dimensional limit theorems for SGD: Effective dynamics and critical scaling

Arous,

Gheissari,

Jagannath

2023

Comm Pure Appl Math

View full text Add to dashboard Cite

We study the scaling limits of stochastic gradient descent (SGD) with constant step‐size in the high‐dimensional regime. We prove limit theorems for the trajectories of summary statistics (i.e., finite‐dimensional functions) of SGD as the dimension goes to infinity. Our approach allows one to choose the summary statistics that are tracked, the initialization, and the step‐size. It yields both ballistic (ODE) and diffusive (SDE) limits, with the limit depending dramatically on the former choices. We show a critical scaling regime for the step‐size, below which the effective ballistic dynamics matches gradient flow for the population loss, but at which, a new correction term appears which changes the phase diagram. About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate. We demonstrate our approach on popular examples including estimation for spiked matrix and tensor models and classification via two‐layer networks for binary and XOR‐type Gaussian mixture models. These examples exhibit surprising phenomena including multimodal timescales to convergence as well as convergence to sub‐optimal solutions with probability bounded away from zero from random (e.g., Gaussian) initializations. At the same time, we demonstrate the benefit of overparametrization by showing that the latter probability goes to zero as the second layer width grows.

show abstract

A generalized Pólya's urn with graph based interactions

Benaı̈m

Benjamini

Chen

et al. 2013

Random Struct Algorithms

Self Cite

View full text Add to dashboard Cite

Given a finite connected graph G, place a bin at each vertex. Two bins are called a pair if they share an edge of G. At discrete times, a ball is added to each pair of bins. In a pair of bins, one of the bins gets the ball with probability proportional to its current number of balls raised by some fixed power α > 0. We characterize the limiting behavior of the proportion of balls in the bins.The proof uses a dynamical approach to relate the proportion of balls to a vector field. Our main result is that the limit set of the proportion of balls is contained in the equilibria set of the vector field. We also prove that if α < 1 then there is a single point v = v(G, α) with non-zero entries such that the proportion converges to v almost surely.A special case is when G is regular and α ≤ 1. We show e.g. that if G is non-bipartite then the proportion of balls in the bins converges to the uniform measure almost surely.

show abstract

Dynamics of stochastic approximation algorithms

Cited by 315 publications

References 33 publications

Learning strict Nash equilibria through reinforcement

Learning strict Nash equilibria through reinforcement

High‐dimensional limit theorems for SGD: Effective dynamics and critical scaling

A generalized Pólya's urn with graph based interactions

Contact Info

Product

Resources

About