Kurdyka–Łojasiewicz–Simon inequality for gradient flows in metric spaces

Hauer, Daniel; Mazón, José M.

doi:10.1090/tran/7801

Cited by 17 publications

(9 citation statements)

References 54 publications

(162 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is clear that convergence of [Θ t ] as t → ∞ is linked to the convergence of the measure η t as t → ∞. At this point, we also mention that convergence of the trajectories [Θ t ] as t → ∞ is linked to the study of Lojasiewicz inequalities [7,27]. To the best of our knowledge no general results in this direction are currently known neither for our case of interest, nor for problems corresponding to the single-layer case (see Appendix C.5 of [10]) (i.e.…”

Section: )mentioning

confidence: 78%

Mean Field Analysis of Neural Networks: A Law of Large Numbers

Sirignano¹,

Spiliopoulos

2020

SIAM J. Appl. Math.

113

View full text Add to dashboard Cite

We analyze multi-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multi-layer neural network output. The limit procedure is valid for any number of hidden layers and it naturally also describes the limiting behavior of the training loss. The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization. The limit satisfies a system of deterministic integro-differential equations. The proof uses methods from weak convergence and stochastic analysis. We show that, under suitable assumptions on the activation functions and the behavior for large times, the limit neural network recovers a global minimum (with zero loss for the objective function). IntroductionMachine learning, and in particular deep learning, has achieved immense practical success, revolutionizing fields such as image, text, and speech recognition. It is also increasingly being used in engineering, medicine, and finance. However, despite their success in practice, there is currently limited mathematical understanding of deep neural networks. This has motivated recent mathematical research on multi-layer learning models such as [39], [40], [41], [20], [21], [42], [49], [50], [43], and [48].Neural networks are nonlinear statistical models whose parameters are estimated from data using stochastic gradient descent (SGD) methods. Deep learning uses neural networks with many layers (i.e., "deep" neural networks), which produces a highly flexible, powerful and effective model in practice. Typically, a neural network with multiple layers between the input and the output layer is called a "deep" neural network, see for example [24]. We analyze multi-layer neural networks that have a fixed number of layers between the input and output layer, and where the number of hidden units in each layer becomes large.Applications of deep learning include image recognition (see [35] and [24]), facial recognition [59], driverless cars [6], speech recognition (see [35], [4], [36], and [60]), and text recognition (see [62] and [57]). Neural networks also find increasing more applications in engineering, robotics, medicine, and finance (see [37], [38], [58], [26], [47], [3], [51], [52], [53], and [54]).In this paper we characterize multi-layer neural networks in the asymptotic regime of large network sizes and large numbers of stochastic gradient descent iterations. We rigorously prove the limit of the neural network output as the number of hidden units increases to infinity. The proof relies upon weak convergence analysis for stochastic processes. The result can be considered a "law of large numbers" for the neural network's output when both the network size and the number of stochastic gradient descent steps grow to infinity. We show that the neural network output in the large hidden-units and large SGD-iterat...

show abstract

Section: )mentioning

confidence: 78%

Mean Field Analysis of Neural Networks: A Law of Large Numbers

Sirignano¹,

Spiliopoulos

2020

SIAM J. Appl. Math.

113

View full text Add to dashboard Cite

show abstract

“…Finally, our local analysis includes a Łojasiewicz inequality in Wasserstein space. Such inequalities were studied in [32,5] for displacement convex functions, which is does not cover our setting.…”

Section: Related Workmentioning

confidence: 99%

Sparse Optimization on Measures with Over-parameterized Gradient Descent

Chizat

2019

Preprint

View full text Add to dashboard Cite

Minimizing a convex function of a measure with a sparsity-inducing penalty is a typical problem arising, e.g., in sparse spikes deconvolution or two-layer neural networks training. We show that this problem can be solved by discretizing the measure and running nonconvex gradient descent on the positions and weights of the particles. For measures on a d-dimensional manifold and under some non-degeneracy assumptions, this leads to a global optimization algorithm with a complexity scaling as log(1/ ) in the desired accuracy , instead of −d for convex methods. The key theoretical tools are a local convergence analysis in Wasserstein space and an analysis of a perturbed mirror descent in the space of measures. Our bounds involve quantities that are exponential in d which is unavoidable under our assumptions.

show abstract

“…• the classical doubly-nonlinear operator ∆ p u m with λ 2 = 1 has been addressed in [9], • the classical 1-Laplace operator has been dealt with in [3,24,25], • decay estimates for the fractional p-Laplacian (−∆ p ) s with s ∈ (0, 1) and p > 1 with λ 2 := 1 have been first established in Section 6 of [16] (see also [15]), • the case of the fractional 1-Laplacian, namely (−∆ p ) s with s ∈ (0, 1) and p := 1 has been treated in [25], • the porous medium equation for λ 2 = 1 has been deeply analyzed in [7], • some interesting estimates for the Kirchoff equation are given in [22], • see also [35], where several decay estimates have been obtained by using integral inequalities.…”

mentioning

confidence: 99%

Decay Estimates in Time for Classical and Anomalous Diffusion

2020

View full text Add to dashboard Cite

We present a series of results focused on the decay in time of solutions of classical and anomalous diffusive equations in a bounded domain. The size of the solution is measured in a Lebesgue space, and the setting comprises time-fractional and space-fractional equations and operators of nonlinear type. We also discuss how fractional operators may affect long-time asymptotics. Decay estimates, methods, results and perspectivesIn this note we present some results, recently obtained in [2,19], focused on the long-time behavior of solutions of evolution equations which may exhibit anomalous diffusion, caused by either time-fractional or space-fractional effects (or both). The case of several nonlinear operators will be also taken into account (and indeed some of the results that we present are new also for classical diffusion run by nonlinear operators).The results that we establish give quantitative bounds on the decay in time of smooth solutions, confined in a smooth bounded set with Dirichlet data. The size of the solution will be measured in classical Lebesgue spaces, and we will detect different types of decays according to the different cases that we take into consid-

show abstract

Kurdyka–Łojasiewicz–Simon inequality for gradient flows in metric spaces

Cited by 17 publications

References 54 publications

Mean Field Analysis of Neural Networks: A Law of Large Numbers

Mean Field Analysis of Neural Networks: A Law of Large Numbers

Sparse Optimization on Measures with Over-parameterized Gradient Descent

Decay Estimates in Time for Classical and Anomalous Diffusion

Contact Info

Product

Resources

About