Efficient computation of limit spectra of sample covariance matrices

Dobriban, Edgar

doi:10.1142/s2010326315500197

Cited by 32 publications

(60 citation statements)

References 39 publications

(127 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Efforts to fit H flexibly from an empirical sample eigenvalue distribution (Li et al, ) emphasize overall fit and are not well suited to test a few large eigenvalues, while a TW test procedure requires only

μ

and

σ

. The recently developed SPECTRODE approach to map a true eigenvalue distribution to the limiting sample eigenvalue distribution (Dobriban, ) is highly efficient, enabling the fitting of a simple parametric model as shown below here. SPECTRODE is fastest when there are few unique true eigenvalues, and we define a simple one parameter probability mass function (pmf)

h_{δ}^{*} false(x false)

, which puts mass

false(1 / 10, 2 / 10, 4 / 10, 2 / 10, 1 / 10 false)

at respective points

false(1 - 2 δ, 1 - δ, 1, 1 + δ, 1 + 2 δ false)

and zero elsewhere.…”

Section: Methodsmentioning

confidence: 99%

Eigenvalue Significance Testing for Genetic Association

Zhou

Marron²,

Wright

2017

Biometrics

View full text Add to dashboard Cite

Genotype eigenvectors are widely used as covariates for control of spurious stratification in genetic association. Significance testing for the accompanying eigenvalues has typically been based on a standard Tracy-Widom limiting distribution for the largest eigenvalue, derived under white-noise assumptions. It is known that even modest local correlation among markers inflates the largest eigenvalues, even in the absence of true stratification. In addition, a few sample eigenvalues may be extreme, creating further complications in accurate testing. We explore several methods to identify appropriate null eigenvalue thresholds, while remaining sensitive to eigenvalues corresponding to population stratification. We introduce a novel block permutation approach, designed to produce an appropriate null eigenvalue distribution by eliminating long-range genomic correlation while preserving local correlation. We also propose a fast approach based on eigenvalue distribution modeling, using a simple fit criterion and the general Marčenko-Pastur equation under a simple discrete eigenvalue model. Block permutation and the model-based approach work well for pure simulations and for data resampled from the 1000 Genomes project. In contrast, we find that the standard approach of computing an “effective” number of markers does not perform well. The performance of the methods is also demonstrated for a motivating example from the International Cystic Fibrosis Consortium.

show abstract

μ

and

σ

h_{δ}^{*} false(x false)

, which puts mass

false(1 / 10, 2 / 10, 4 / 10, 2 / 10, 1 / 10 false)

at respective points

false(1 - 2 δ, 1 - δ, 1, 1 + δ, 1 + 2 δ false)

and zero elsewhere.…”

Section: Methodsmentioning

confidence: 99%

Eigenvalue Significance Testing for Genetic Association

Zhou

Marron²,

Wright

2017

Biometrics

View full text Add to dashboard Cite

show abstract

“…Our new method of derandomized PA chooses k factors where k satisfies

σ_{k}^{2} false(n^{- 1 false/ 2} X false) > U false(F_{p false/ n, \overset{D}{true}} false) ⩾ σ_{k + 1}^{2} false(n^{- 1 false/ 2} X false) .

The spectrode method (Dobriban, ) computes

F_{p false/ n, \overset{D}{true}}

from which we can obtain this upper edge. We give an even faster algorithm in Section 7.1 that computes

U (F_{p / n, \overset{false^}{D}})

directly.…”

Section: Derandomizationmentioning

confidence: 99%

“…We describe a fast method to compute the upper edge

U (F_{γ, H})

of the MP distribution, given the population spectrum H and the aspect ratio γ . Our method described here is, implicitly, one of the steps of the spectrode method, which computes the density of the whole MP distribution F γ , H (Dobriban, ). However, this subproblem was not considered separately in Dobriban (), and thus calling the spectrode method can be inefficient when we need only the upper edge.…”

Section: Computing the Thresholdsmentioning

confidence: 99%

See 1 more Smart Citation

Deterministic Parallel Analysis: An Improved Method for Selecting Factors and Principal Components

Dobriban

Owen

2018

Journal of the Royal Statistical Society Series B: Statistical Methodology

Self Cite

View full text Add to dashboard Cite

Summary Factor analysis and principal component analysis are used in many application areas. The first step, choosing the number of components, remains a serious challenge. Our work proposes improved methods for this important problem. One of the most popular state of the art methods is parallel analysis (PA), which compares the observed factor strengths with simulated strengths under a noise‐only model. The paper proposes improvements to PA. We first derandomize it, proposing deterministic PA, which is faster and more reproducible than PA. Both PA and deterministic PA are prone to a shadowing phenomenon in which a strong factor makes it difficult to detect smaller but more interesting factors. We propose deflation to counter shadowing. We also propose to raise the decision threshold to improve estimation accuracy. We prove several consistency results for our methods, and test them in simulations. We also illustrate our methods on data from the human genome diversity project, where they significantly improve the accuracy.

show abstract

“…Much of Zhou et al () was concerned with direct modelling of the true eigenvalue distribution using the general Marc̆enko–Pastur (MP) law, for which a difficult computational inverse problem was made faster by the SPECTRODE algorithm (Dobriban, ). Once an appropriate fit to the MP distribution was determined, the corresponding TW approximation was used for testing significance of sample eigenvalues.…”

Section: Introductionmentioning

confidence: 99%

A note on cyclic shift permutation testing for large eigenvalues

Zhou

2019

Stat

View full text Add to dashboard Cite

Recent publications have described the problem of testing for the “significance” of large sample (empirical) matrix eigenvalues in the presence of modest variation of underlying true eigenvalues. This modest variation often can be ascribed to endemic dependence in one matrix dimension (e.g., rows), whereas the null hypothesis concerns the other dimension (columns). The need for such testing frequently arises in genomics, time‐series analysis, and a variety of other fields. However, the tools available for testing are underdeveloped, with statistical properties that may be sensitive to the true eigenvalues. The purpose of this note is to point the reader to this emerging literature and to suggest that the tool of cyclic shift permutation may be well‐suited to the problem.

show abstract

Efficient computation of limit spectra of sample covariance matrices

Cited by 32 publications

References 39 publications

Eigenvalue Significance Testing for Genetic Association

Eigenvalue Significance Testing for Genetic Association

Deterministic Parallel Analysis: An Improved Method for Selecting Factors and Principal Components

A note on cyclic shift permutation testing for large eigenvalues

Contact Info

Product

Resources

About