Sparse Optimization on Measures with Over-parameterized Gradient Descent

Chizat, Lenaic

doi:10.48550/arxiv.1907.10300

Cited by 18 publications

(58 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 Our arguments can indeed be applied to situations where the parameter space can be factored as R+ × Θ where Θ is a compact Riemannian manifold without boundary, see Chizat (2019). For clarity, we limit ourselves to a parameter space R p (which corresponds to Θ = S p−1 ) while for S-ReLU, this would correspond to Θ = S p−1 S p−1 .…”

Section: -Homogeneous Neural Networkmentioning

confidence: 99%

“…A general drawback of those mean-field analyses is that they are mostly non-quantitative, both in terms of number of neurons and number of iterations. While some works have shown quantitative results by modifying the dynamics (Mei et al, 2019;Wei et al, 2019;Chizat, 2019), we do not take this path in order to stay close to the way neural networks are used in practice and because our numerical experiments suggest that those modifications are not necessary to obtain a good practical behavior. Finally, we stress that our analysis does not take place in the lazy training regime which consists of training dynamics that can be analyzed in a perturbative regime around the initialization (see, e.g., Li and Liang, 2018;Jacot et al, 2018;Du et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

Chizat¹,

Bach²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. Towards understanding this phenomenon, we analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations. We show that the limits of the gradient flow on exponentially tailed losses can be fully characterized as a max-margin classifier in a certain non-Hilbertian space of functions. In presence of hidden low-dimensional structures, the resulting margin is independent of the ambiant dimension, which leads to strong generalization bounds. In contrast, training only the output layer implicitly solves a kernel support vector machine, which a priori does not enjoy such an adaptivity. Our analysis of training is non-quantitative in terms of running time but we prove computational guarantees in simplified settings by showing equivalences with online mirror descent. Finally, numerical experiments suggest that our analysis describes well the practical behavior of two-layer neural networks with ReLU activation and confirm the statistical benefits of this implicit bias.

show abstract

Section: -Homogeneous Neural Networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

Chizat¹,

Bach²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Firstly, to ensure the full support assumption, we can consider extending the neural birthdeath [16,51] to deep ResNets. Neural birth-death dynamics considers the gradient flow in the Wasserstein-Fisher-Rao space [19] rather than the Wasserstein space and ensures convergence.…”

Section: Discussionmentioning

confidence: 99%

A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth

Lu¹,

Ma²,

Lu³

et al. 2020

Preprint

View full text Add to dashboard Cite

Training deep neural networks with stochastic gradient descent (SGD) can often achieve zero training loss on real-world tasks although the optimization landscape is known to be highly non-convex. To understand the success of SGD for training deep neural networks, this work presents a mean-field analysis of deep residual networks, based on a line of works that interpret the continuum limit of the deep residual network as an ordinary differential equation when the network capacity tends to infinity. Specifically, we propose a new continuum limit of deep residual networks, which enjoys a good landscape in the sense that every local minimizer is global. This characterization enables us to derive the first global convergence result for multilayer neural networks in the mean-field regime. Furthermore, without assuming the convexity of the loss landscape, our proof relies on a zero-loss assumption at the global minimizer that can be achieved when the model shares a universal approximation property. Key to our result is the observation that a deep residual network resembles a shallow network ensemble [59], i.e. a two-layer network. We bound the difference between the shallow network and our ResNet model via the adjoint sensitivity method, which enables us to apply existing mean-field analyses of two-layer networks to deep networks. Furthermore, we propose several novel training schemes based on the new continuous model, including one training procedure that switches the order of the residual blocks and results in strong empirical performance on the benchmark datasets.

show abstract

“…While there have been several proven global convergence guarantees for multilayer networks [17,21,20], understanding of the convergence rate is still lacking. Even in the shallow case, global convergence has been studied only for a type of sparsity-inducing regularization [5,4]. Unless the convergence rate for multilayer networks is generally perilous (an unlikely scenario in light of the experiments in [15]), our result is expected to be relevant.…”

Section: Introductionmentioning

confidence: 86%

Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training

Pham¹,

Nguyen²

2021

Preprint

View full text Add to dashboard Cite

The mean field theory of multilayer neural networks centers around a particular infinite-width scaling, in which the learning dynamics is shown to be closely tracked by the mean field limit. A random fluctuation around this infinite-width limit is expected from a large-width expansion to the next order. This fluctuation has been studied only in the case of shallow networks, where previous works employ heavily technical notions or additional formulation ideas amenable only to that case. Treatment of the multilayer case has been missing, with the chief difficulty in finding a formulation that must capture the stochastic dependency across not only time but also depth. In this work, we initiate the study of the fluctuation in the case of multilayer networks, at any network depth. Leveraging on the neuronal embedding framework recently introduced by Nguyen and Pham [17], we systematically derive a system of dynamical equations, called the second-order mean field limit, that captures the limiting fluctuation distribution. We demonstrate through the framework the complex interaction among neurons in this second-order mean field limit, the stochasticity with cross-layer dependency and the nonlinear time evolution inherent in the limiting fluctuation. A limit theorem is proven to relate quantitatively this limit to the fluctuation realized by large-width networks. We apply the result to show a stability property of gradient descent mean field training: in the large-width regime, along the training trajectory, it progressively biases towards a solution with "minimal fluctuation" (in fact, vanishing fluctuation) in the learned output function, even after the network has been initialized at or has converged (sufficiently fast) to a global optimum. This extends a similar phenomenon previously shown only for shallow networks with a squared loss in the empirical risk minimization setting, to multilayer networks with a loss function that is not necessarily convex in a more general setting.

show abstract

Sparse Optimization on Measures with Over-parameterized Gradient Descent

Cited by 18 publications

References 37 publications

Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss

A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth

Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training

Contact Info

Product

Resources

About