Direct Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices

Abstract

Standard-Nutzungsbedingungen:Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden.Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen.Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Abstract This paper introduces a nonlinear shrinkage estimator of the covariance matrix that does not require recovering the population eigenvalues first. We estimate the sample spectral density and its Hilbert transform directly by smoothing the sample eigenvalues with a variable-bandwidth kernel. Relative to numerically inverting the so-called QuEST function, the main advantages of direct kernel estimation are: (1) it is much easier to comprehend because it is analogous to kernel density estimation; (2) it is only twenty lines of code in Matlab -as opposed to thousands -which makes it more verifiable and customizable; (3) it is 200 times faster without significant loss of accuracy; and (4) it can handle matrices of a dimension larger by a factor of ten. Even for dimension 10, 000, the code runs in less than two minutes on a desktop computer; this makes the power of nonlinear shrinkage as accessible to applied statisticians as the one of linear shrinkage. Terms of use: Documents in

Analytic approach to variance optimization under an ℓ1 constraint

“…A common regularizer is 2 (shrinkage or ridge regression) whose effect has been studied by a number of authors, see [1][2][3][4][5][6][7] among many others. In its most recent, nonlinear form shrinkage can produce very good quality estimates [8][9][10]. Another popular regularizer is based on the 1 norm (lasso) [11].…”

G-TwYST harmonisation of statistical methods for use of omics data in food safety assessment

“…However, when these variances are dispersed (e.g. much larger variation can be observed along a few axis of the ellipse), linear shrinkage only improves upon the sample estimator of the covariance matrix slightly Ledoit and Wolf 2012;Ledoit and Wolf 2017;Ledoit and Wolf 2018]. Note that this is similar to the low-dimensional case that was considered in the previous subsection.…”

“…Unfortunately, most nonlinear shrinkage estimators are computationally quite expensive and therefore not applicable to data of moderate to high-dimensionality. Recently, however, Ledoit and Wolf introduced a nonlinear shrinkage estimator of the covariance matrix that is much faster than previous approaches without loss of numerical accuracy [Ledoit and Wolf 2017]. Here, we consider this nonlinear shrinkage approach in combination with Hoteling's T 2 for difference testing between a GM and a set of reference varieties.…”

Early deviation from normal structural connectivity

“…Mindful of the large number of tract bundles (22) relative to the number of controls (31) and patients (34), we used conservative approximations of C in place of the conventional Pearson correlation. Shrinkage estimators 11 remove potentially spurious correlations in small datasets. To ensure robustness, we used a subset of 25 randomly selected controls to compute the covariance and permuted this 1,000 times.…”