Surprises in high-dimensional ridgeless least squares interpolation

Hastie, Trevor; Montanari, Andrea; Rosset, Saharon; Tibshirani, Ryan J.

doi:10.1214/21-aos2133

Cited by 202 publications

(124 citation statements)

References 37 publications

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…The work of Fan and Lv 67 focused on variable selection and the asymptotic behavior of their novel iterative sure independence screening under varying high‐dimensional or ultrahigh‐dimensional settings. During the preparation of the current version (the first draft was uploaded to arXiv in 2017 69 ), it has been found that the optimal ridge penalty is positive when the coefficients are generated from a distribution 32,33 whereas the optimal ridge penalty could be negative when the coefficients are fixed 34 . These studies rely on random matrix theorems and assumptions of the covariance structure of the explanatory variable.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Ridge‐penalized adaptive Mantel test and its application in imaging genetics

Pluta

Tong

Xue

et al. 2021

Statistics in Medicine

View full text Add to dashboard Cite

We propose a ridge-penalized adaptive Mantel test (AdaMant) for evaluating the association of two high-dimensional sets of features. By introducing a ridge penalty, AdaMant tests the association across many metrics simultaneously. We demonstrate how ridge penalization bridges Euclidean and Mahalanobis distances and their corresponding linear models from the perspective of association measurement and testing. This result is not only theoretically interesting but also has important implications in penalized hypothesis testing, especially in high dimensional settings such as imaging genetics. Applying the proposed method to an imaging genetic study of visual working memory in health adults, we identified interesting associations of brain connectivity (measured by EEG coherence) with selected genetic features.

show abstract

Section: Discussionmentioning

confidence: 99%

“…For prediction in deep learning and estimation, the results are mixed. For example, the optimal ridge penalty is found to be positive when the coefficients are generated from a distribution 32,33 whereas the optimal ridge penalty might be negative when the coefficients are fixed 34 . However, in practice, the true model is never known.…”

Section: Introductionmentioning

confidence: 99%

Ridge‐penalized adaptive Mantel test and its application in imaging genetics

Pluta

Tong

Xue

et al. 2021

Statistics in Medicine

View full text Add to dashboard Cite

show abstract

“…In particular, impressive results in image classification, pattern recognition and feature extraction were achieved by means of deep convolutional neural networks. Borrowing from Wigner, the efforts of many researchers are currently directed to provide explanations for the "unreasonable effectiveness" of these models and related intriguing phenomena, such as the double descent error curve [3,12,21,23] or the instability to adversarial attacks [1,8,10,14,28].…”

Section: Introductionmentioning

confidence: 99%

Stability of the scattering transform for deformations with minimal regularity

Nicola¹,

Trapasso²

2022

Preprint

View full text Add to dashboard Cite

Within the mathematical analysis of deep convolutional neural networks, the wavelet scattering transform introduced by Stéphane Mallat is a unique example of how the ideas of multiscale analysis can be combined with a cascade of modulus nonlinearities to build a nonexpansive, translation invariant signal representation with provable geometric stability properties, namely Lipschitz continuity to the action of small C 2 diffeomorphisms -a remarkable result for both theoretical and practical purposes, inherently depending on the choice of the filters and their arrangement into a hierarchical architecture. In this note, we further investigate the intimate relationship between the scattering structure and the regularity of the deformation in the Hölder regularity scale C α , α > 0. We are able to precisely identify the stability threshold, proving that stability is still achievable for deformations of class C α , α > 1, whereas instability phenomena can occur at lower regularity levels modelled by C α , 0 ≤ α < 1. While the behaviour at the threshold given by Lipschitz (or even C 1 ) regularity remains beyond reach, we are able to prove a stability bound in that case, up to ε losses.

show abstract

“…The most representative approach involves the utilization of sparsity via ℓ 1 -norm regularization and its variants (Candes and Tao, 2007;Van de Geer, 2008;Bühlmann and Van De Geer, 2011;Hastie et al, 2019), which is effective when a signal to be estimated has many zero elements. Another study investigated the high-dimensional limit of risks of estimators (Dobriban and Wager, 2018;Belkin et al, 2019;Hastie et al, 2022;Bartlett et al, 2020) and revealed the risk limit when 𝑛 data instances and 𝑝 parameters diverged infinitely, while their ratio 𝑝/𝑛 converged to a positive constant value. Interpolators are a typical estimator that perfectly fit the observed data under the 𝑝 𝑛 setting; moreover, it has been demonstrated that the risk or Date: April 19, 2022.…”

mentioning

confidence: 99%

“…variance of interpolators converges to zero (Liang and Rakhlin, 2020;Hastie et al, 2022;Ba et al, 2019). It should be noted that these studies did not use the sparsity of data or signals; however, they are compatible with recent large-scale data analyses.…”

mentioning

confidence: 99%

Benign Overfitting in Time Series Linear Model with Over-Parameterization

Nakakita¹,

Imaizumi²

2022

Preprint

View full text Add to dashboard Cite

A. The success of large-scale models in recent years has increased the importance of statistical models with numerous parameters. Several studies have analyzed over-parameterized linear models with high-dimensional data that may not be sparse; however, existing results depend on the independent setting of samples. In this study, we analyze a linear regression model with dependent time series data under over-parameterization settings. We consider an estimator via interpolation and developed a theory for excess risk of the estimator under multiple dependence types. This theory can treat infinite-dimensional data without sparsity and handle long-memory processes in a unified manner. Moreover, we bound the risk in our theory via the integrated covariance and nondegeneracy of autocorrelation matrices. The results show that the convergence rate of risks with short-memory processes is identical to that of cases with independent data, while long-memory processes slow the convergence rate. We also present several examples of specific dependent processes that can be applied to our setting.

show abstract

Surprises in high-dimensional ridgeless least squares interpolation

Cited by 202 publications

References 37 publications

Ridge‐penalized adaptive Mantel test and its application in imaging genetics

Ridge‐penalized adaptive Mantel test and its application in imaging genetics

Stability of the scattering transform for deformations with minimal regularity

Benign Overfitting in Time Series Linear Model with Over-Parameterization

Contact Info

Product

Resources

About