Asymptotics of eigenstructure of sample correlation matrices for high-dimensional spiked models

Morales-Jiménez, David; Johnstone, Iain M.; McKay, Matthew R.; Yang, Jeha

doi:10.5705/ss.202019.0052

Cited by 21 publications

(21 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is straightforward, so we omit the detail. Given a data set, the spikes of the population correlation R have to be determined; see Fan et al (2020) and Morales-Jimenez et al (2021).…”

Section: One Sample Mean Test For Large P and Small Nmentioning

confidence: 99%

Mean Test with Fewer Observation than Dimension and Ratio Unbiased Estimator for Correlation Matrix

Jiang,

2021

Preprint

View full text Add to dashboard Cite

Hotelling's T-squared test is a classical tool to test if the normal mean of a multivariate normal distribution is a specified one or the means of two multivariate normal means are equal. When the population dimension is higher than the sample size, the test is no longer applicable. Under this situation, in this paper we revisit the tests proposed by Srivastava and Du (2008), who revise the Hotelling's statistics by replacing Wishart matrices with their diagonal matrices. They show the revised statistics are asymptotically normal. We use the random matrix theory to examine their statistics again and find that their discovery is just part of the big picture. In fact, we prove that their statistics, decided by the Euclidean norm of the population correlation matrix, can go to normal, mixing chi-squared distributions and a convolution of both.Examples are provided to show the phase transition phenomenon between the normal and mixing chi-squared distributions. The second contribution of ours is a rigorous derivation of an asymptotic ratio-unbiased-estimator of the squared Euclidean norm of the correlation matrix.

show abstract

“…It is straightforward, so we omit the detail. Given a data set, the spikes of the population correlation R have to be determined; see Fan et al (2020) and Morales-Jimenez et al (2021).…”

Section: One Sample Mean Test For Large P and Small Nmentioning

confidence: 99%

Mean Test with Fewer Observation than Dimension and Ratio Unbiased Estimator for Correlation Matrix

Jiang,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Whether such an approach is statistically beneficial is task-dependent and deserves a more thorough consideration in future work. On the one hand, too many nodes may induce many errors and a systematic distortion in the spectrum of empirical covariance matrices (Donoho et al, 2013, Morales-Jimenez et al, 2021, on the other hand single errors have higher impact in smaller networks, and node estimation constitutes yet another challenging task in the network construction pipeline. A key question in this regard is: How can we minimize the edge-wise estimation variance in the downsized network?…”

Section: Discussionmentioning

confidence: 99%

Pitfalls of Climate Network Construction: A Statistical Perspective

Haas¹,

Goswami²,

Luxburg³

2022

Preprint

View full text Add to dashboard Cite

Network-based analyses of dynamical systems have become increasingly popular in climate science. Here we address network construction from a statistical perspective and highlight the often ignored fact that the calculated correlation values are only empirical estimates. To measure spurious behaviour as deviation from a ground truth network, we simulate time-dependent isotropic random fields on the sphere and apply common network construction techniques. We find several ways in which the uncertainty stemming from the estimation procedure has major impact on network characteristics. When the data has locally coherent correlation structure, spurious link bundle teleconnections and spurious high-degree clusters have to be expected. Anisotropic estimation variance can also induce severe biases into empirical networks. We validate our findings with ERA5 reanalysis data. Moreover we explain why commonly applied resampling procedures are inappropriate for significance evaluation and propose a statistically more meaningful ensemble construction framework. By communicating which difficulties arise in estimation from scarce data and by presenting which design decisions increase robustness, we hope to contribute to more reliable climate network construction in the future.

show abstract

“…[20]. We however notice that papers addressing the behaviour of the corresponding sample correlation matrices are quite scarce, see [21] when the low rank perturbation affects only the first components of the observations.…”

Section: A Low Vs High-dimensional Regimementioning

confidence: 95%

On the Detection of Low-Rank Signal in the Presence of Spatially Uncorrelated Noise: A Frequency Domain Approach

Rosuel

Vallet

Loubaton

et al. 2021

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

This paper analyzes the detection of a M -dimensional useful signal modeled as the output of a M ×K MIMO filter driven by a K-dimensional white Gaussian noise, and corrupted by a M -dimensional Gaussian noise with mutually uncorrelated components. The study is focused on frequency domain test statistics based on the eigenvalues of an estimate of the spectral coherence matrix (SCM), obtained as a renormalization of the frequency-smoothed periodogram of the observed signal. If N denotes the sample size and B the smoothing span, it is proved that in the high-dimensional regime where M, B, N converge to infinity while K remains fixed, the SCM behaves as a certain correlated Wishart matrix. Exploiting well-known results on the behaviour of the eigenvalues of such matrices, it is deduced that the standard tests based on linear spectral statistics of the SCM fail to detect the presence of the useful signal in the high-dimensional regime. A new test based on the SCM, which is proved to be consistent, is also proposed, and its statistical performance is evaluated through numerical simulations.

show abstract

Asymptotics of eigenstructure of sample correlation matrices for high-dimensional spiked models

Cited by 21 publications

References 42 publications

Mean Test with Fewer Observation than Dimension and Ratio Unbiased Estimator for Correlation Matrix

Mean Test with Fewer Observation than Dimension and Ratio Unbiased Estimator for Correlation Matrix

Pitfalls of Climate Network Construction: A Statistical Perspective

On the Detection of Low-Rank Signal in the Presence of Spatially Uncorrelated Noise: A Frequency Domain Approach

Contact Info

Product

Resources

About