Condition-Number-Regularized Covariance Estimation

Abstract

Estimation of high-dimensional covariance matrices is known to be a difficult problem, has many applications, and is of current interest to the larger statistics community. In many applications including so-called the “large p small n” setting, the estimate of the covariance matrix is required to be not only invertible, but also well-conditioned. Although many regularization schemes attempt to do this, none of them address the ill-conditioning problem directly. In this paper, we propose a maximum likelihood approach, with the direct goal of obtaining a well-conditioned estimator. No sparsity assumption on either the covariance matrix or its inverse are are imposed, thus making our procedure more widely applicable. We demonstrate that the proposed regularization scheme is computationally efficient, yields a type of Steinian shrinkage estimator, and has a natural Bayesian interpretation. We investigate the theoretical properties of the regularized covariance estimator comprehensively, including its regularization path, and proceed to develop an approach that adaptively determines the level of regularization that is required. Finally, we demonstrate the performance of the regularized estimator in decision-theoretic comparisons and in the financial portfolio optimization setting. The proposed approach has desirable properties, and can serve as a competitive procedure, especially when the sample size is small and when a well-conditioned estimator is required.

Design-free estimation of variance matrices

“…An alternative approach whereby offdiagonal elements are downweighted towards zero is given in McMurry and Politis (2010) and Politis (2011) in the context of time series. See also an approach to shrinkage via condition-number regularization in Won, Lim, Kim, and Rajaratnam (2013).…”

Simulation of the ozone pretreatment of wheat straw

“…where k max is the largest eigenvalue of the covariance matrix and k min is the smallest eigenvalue of the covariance matrix, is a measure of the numerical stability of the covariance matrix (Won et al, 2013), where a lower value signifies a well-conditioned matrix. The eigenvalues of any square matrix can be directly computed in MATLAB.…”

A note on the lack of symmetry in the graphical lasso

“…A third possible indicator is when the condition number of the resulting estimate is too high. Recently, Won et al (2012) provide impetus for constraining the condition number of the covariance matrix; in light of that work, the condition number can perhaps be used as a guide in choosing λ. 6…”