The Spectrum of Random Kernel Matrices: Universality Results for Rough and Varying Kernels

Do, Yen; Vu, Van

doi:10.1142/s2010326313500056

Cited by 28 publications

(18 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, the leave-one-out method can be used to derive a recursion for the resolvent, as first shown for this type of matrices in Cheng and Singer (2013), and the moments method was first used in Fan and Montanari (2019) (both of these papers consider symmetric random matrices, but these techniques extend to the asymmetric case). Further results on kernel random matrices can be found in Do and Vu (2013), Louart, Liao and Couillet (2018) and Pennington and Worah (2018).…”

Section: Proportional Scalingmentioning

confidence: 94%

Deep learning: a statistical viewpoint

2021

View full text Add to dashboard Cite

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behaviour of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favourable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

show abstract

Section: Proportional Scalingmentioning

confidence: 94%

Deep learning: a statistical viewpoint

2021

View full text Add to dashboard Cite

show abstract

“…This result was generalized by Do and Vu to the setting of non-Gaussian entries x ij in [23]. Before stating our main results, let us discuss some basic properties of this limit measure: For a linear kernel function k(x) = ax, µ a,ν,γ is a translation and rescaling of the Marcenko-Pastur law.…”

Section: Introductionmentioning

confidence: 98%

The spectral norm of random inner-product kernel matrices

Fan

Montanari

2018

Probab. Theory Relat. Fields

View full text Add to dashboard Cite

We study an "inner-product kernel" random matrix model, whose empirical spectral distribution was shown by Xiuyuan Cheng and Amit Singer to converge to a deterministic measure in the large n and p limit. We provide an interpretation of this limit measure as the additive free convolution of a semicircle law and a Marcenko-Pastur law. By comparing the tracial moments of this random matrix to those of a deformed GUE matrix with the same limiting spectrum, we establish that for odd kernel functions, the spectral norm of this matrix convergences almost surely to the edge of the limiting spectrum. Our study is motivated by the analysis of a covariance thresholding procedure for the statistical detection and estimation of sparse principal components, and our results characterize the limit of the largest eigenvalue of the thresholded sample covariance matrix in the null setting.

show abstract

“…In particular, the leave-one-out method can be used to derive a recursion for the resolvent, as first shown for this type of matrices in [CS13], and the moments method was first used in [FM19] (both of these papers consider symmetric random matrices, but these techniques extend to the asymmetric case). Further results on kernel random matrices can be found in [DV13,LLC18] and [PW18].…”

Section: Proportional Scalingmentioning

confidence: 99%

Deep learning: a statistical viewpoint

Montanari¹,

Rakhlin²

2021

Preprint

View full text Add to dashboard Cite

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behavior of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favorable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

show abstract

The Spectrum of Random Kernel Matrices: Universality Results for Rough and Varying Kernels

Cited by 28 publications

References 31 publications

Deep learning: a statistical viewpoint

Deep learning: a statistical viewpoint

The spectral norm of random inner-product kernel matrices

Deep learning: a statistical viewpoint

Contact Info

Product

Resources

About