Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Mignacco, Francesca; Urbani, Pierfrancesco; Zdeborová, Lenka

doi:10.1088/2632-2153/ac0615

Cited by 18 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…tensor PCA [57,58], phase retrieval [59], and high-dimensional linear classifiers [60][61][62][63], but has yet to be developed for deep feature learning. By developing a self-consistent DMFT of deep NNs, we gain insight into how features evolve in the rich regime of network training, while retaining many pleasant analytic properties of the infinite width limit.…”

Section: Related Workmentioning

confidence: 99%

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Bordelon¹,

Pehlevan²

2022

Preprint

View full text Add to dashboard Cite

We analyze feature learning in infinite width neural networks trained with gradient flow through a self-consistent dynamical field theory. We construct a collection of deterministic dynamical order parameters which are inner-product kernels for hidden unit activations and gradients in each layer at pairs of time points, providing a reduced description of network activity through training. These kernel order parameters collectively define the hidden layer activation distribution, the evolution of the neural tangent kernel, and consequently output predictions. For deep linear networks, these kernels satisfy a set of algebraic matrix equations. For nonlinear networks, we provide an alternating sampling procedure to self-consistently solve for the kernel order parameters. We provide comparisons of the self-consistent solution to various approximation schemes including the static NTK approximation, gradient independence assumption, and leading order perturbation theory, showing that each of these approximations can break down in regimes where general selfconsistent solutions still provide an accurate description. Lastly, we provide experiments in more realistic settings which demonstrate that the loss and kernel dynamics of CNNs at fixed feature learning strength is preserved across different widths on a CIFAR classification task.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Bordelon¹,

Pehlevan²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This is a powerful formalism but the integral equations must usually be solved entirely numerically which itself is not a trivial task. For problems close to the present context (neural networks,generalized linear models, phase retrieval) DMFT has been developed in the recent works [42,43,44,45,46]. We think that comprehensively comparing this formalism with the present approach is an interesting open problem.…”

Section: Discussionmentioning

confidence: 96%

Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model

Bodin¹,

Macris²

2021

Preprint

View full text Add to dashboard Cite

Recent evidence has shown the existence of a so-called double-descent and even triple-descent behavior for the generalization error of deep-learning models. This important phenomenon commonly appears in implemented neural network architectures, and also seems to emerge in epoch-wise curves during the training process. A recent line of research has highlighted that random matrix tools can be used to obtain precise analytical asymptotics of the generalization (and training) errors of the random feature model. In this contribution, we analyze the whole temporal behavior of the generalization and training errors under gradient flow for the random feature model. We show that in the asymptotic limit of large system size the full time-evolution path of both errors can be calculated analytically. This allows us to observe how the double and triple descents develop over time, if and when early stopping is an option, and also observe time-wise descent structures. Our techniques are based on Cauchy complex integral representations of the errors together with recent random matrix methods based on linear pencils.

show abstract

“…Unfortunately, our solution algorithms display convergence problems even at rather short times, thus preventing us to obtain reliable numerical predictions for the jamming transition from DMFT. Similar problems where encountered in reference [17]. To obtain more insight into the problem, we then simulated the RLG in several dimensions ranging from d = 2 to d = 22.…”

Section: Discussionmentioning

confidence: 99%

“…The unknown function f (x) is then parametrized by a guess function g(x; θ), whose parameters θ can be learnt by minimizing a loss function accounting for the error made in labelling data from the training set. Also in this case the GD dynamics can be surprisingly complex [10][11][12][13][14][15][16][17].…”

Section: Introductionmentioning

confidence: 99%

Gradient descent dynamics and the jamming transition in infinite dimensions

Manacorda¹,

Zamponi²

2022

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

Gradient descent dynamics in complex energy landscapes, i.e. featuring multiple minima, finds application in many different problems, from soft matter to machine learning. Here, we analyze one of the simplest examples, namely that of soft repulsive particles in the limit of infinite spatial dimension d. The gradient descent dynamics then displays a jamming transition: at low density, it reaches zero-energy states in which particles’ overlaps are fully eliminated, while at high density the energy remains finite and overlaps persist. At the transition, the dynamics becomes critical. In the d → ∞ limit, a set of self-consistent dynamical equations can be derived via mean field theory. We analyze these equations and we present some partial progress towards their solution. We also study the random Lorentz gas in a range of d = 2…22, and obtain a robust estimate for the jamming transition in d → ∞. The jamming transition is analogous to the capacity transition in supervised learning, and in the appendix we discuss this analogy in the case of a simple one-layer fully-connected perceptron.

show abstract

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Cited by 18 publications

References 17 publications

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model

Gradient descent dynamics and the jamming transition in infinite dimensions

Contact Info

Product

Resources

About