Continuous-time Models for Stochastic Optimization Algorithms

Orvieto, Antonio; Lucchi, Aurelien

doi:10.48550/arxiv.1810.02565

Cited by 4 publications

(8 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, the authors use backward error analysis to study how close SGD is to its approximation using a high order (involving the Hessian) stochastic modified equation. It would be interesting to derive a similar result for a stochastic variant of GD-ODE, such as the one studied in [40].…”

Section: Discussionmentioning

confidence: 97%

Shadowing Properties of Optimization Algorithms

Orvieto¹,

Lucchi²

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Ordinary differential equation (ODE) models of gradient-based optimization methods can provide insights into the dynamics of learning and inspire the design of new algorithms. Unfortunately, this thought-provoking perspective is weakened by the fact that -in the worst case -the error between the algorithm steps and its ODE approximation grows exponentially with the number of iterations. In an attempt to encourage the use of continuous-time methods in optimization, we show that, if some additional regularity on the objective is assumed, the ODE representations of Gradient Descent and Heavy-ball do not suffer from the aforementioned problem, once we allow for a small perturbation on the algorithm initial condition. In the dynamical systems literature, this phenomenon is called shadowing. Our analysis relies on the concept of hyperbolicity, as well as on tools from numerical analysis.

show abstract

Section: Discussionmentioning

confidence: 97%

Shadowing Properties of Optimization Algorithms

Orvieto¹,

Lucchi²

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…which is inspired by the Lyapunov function in [22] (end of page 22). Differentiating, using Lemma 12 and α-weakly-quasi-convexity, we get the result.…”

Section: Discussionmentioning

confidence: 99%

A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization

Alimisis,

Orvieto,

Bécigneul

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below. This ODE can be seen as a generalization of the second-order ODE derived for Euclidean spaces and can also serve as an analysis tool. We analyze the convergence behavior of this ODE for different types of functions, such as geodesically convex, strongly-convex and weakly-quasi-convex. We demonstrate how such an ODE can be discretized using a semiimplicit and Nesterov-inspired numerical integrator, that empirically yields stable algorithms which are faithful to the continuoustime analysis and exhibit accelerated convergence.

show abstract

“…In lieu of the inability of convex optimization theory to explain the behavior of SGD in non-convex settings, it is common to consider the behavior of Markov process models for stochastic optimizers [38]. These models are often continuous for ease of analysis [50,58], although discrete-time treatments have become increasingly popular [16,28]. Such continuous-time models are formulated as stochastic differential equations dW t " µpW t qdt `σpW t qdX t , where X t is typically Brownian motion, or some other Lèvy process, and derived through the (generalized) central limit theorem and taking learning rates to zero [23].…”

Section: Optimization-based Generalizationmentioning

confidence: 99%

“…A precise link to stochastic optimization comes through the following observation: suppose that W t , t P r0, 1s, is a continuous-time Markov process with transition kernel P t px, Eq, e.g., a continuous-time model of a stochastic optimizer [50]. Combining [65, Theorem 4.1] and [45,Theorem 5.7], if W t is spatially homogeneous, and…”

Section: Corollary 1 Suppose That W Has Hausdorff Dimension α and Is ...mentioning

confidence: 99%

See 1 more Smart Citation

Generalization Properties of Stochastic Optimizers via Trajectory Analysis

Hodgkinson¹,

Şimşekli²,

Khanna³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite the ubiquitous use of stochastic optimization algorithms in machine learning, the precise impact of these algorithms on generalization performance in realistic non-convex settings is still poorly understood. In this paper, we provide an encompassing theoretical framework for investigating the generalization properties of stochastic optimizers, which is based on their dynamics. We first prove a generalization bound attributable to the optimizer dynamics in terms of the celebrated Fernique-Talagrand functional applied to the trajectory of the optimizer. This data-and algorithm-dependent bound is shown to be the sharpest possible in the absence of further assumptions. We then specialize this result by exploiting the Markovian structure of stochastic optimizers, deriving generalization bounds in terms of the (data-dependent) transition kernels associated with the optimization algorithms. In line with recent work that has revealed connections between generalization and heavy-tailed behavior in stochastic optimization, we link the generalization error to the local tail behavior of the transition kernels. We illustrate that the local power-law exponent of the kernel acts as an effective dimension, which decreases as the transitions become "less Gaussian." We support our theory with empirical results from a variety of neural networks, and we show that both the Fernique-Talagrand functional and the local power-law exponent are predictive of generalization performance.

show abstract

Continuous-time Models for Stochastic Optimization Algorithms

Cited by 4 publications

References 30 publications

Shadowing Properties of Optimization Algorithms

Shadowing Properties of Optimization Algorithms

A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization

Generalization Properties of Stochastic Optimizers via Trajectory Analysis

Contact Info

Product

Resources

About