Shrinkage Algorithms for MMSE Covariance Estimation

Abstract

We address covariance estimation in the sense of minimum mean-squared error (MMSE) for Gaussian samples. Specifically, we consider shrinkage methods which are suitable for high dimensional problems with a small number of samples (large p small n). First, we improve on the Ledoit-Wolf (LW) method by conditioning on a sufficient statistic. By the Rao-Blackwell theorem, this yields a new estimator called RBLW, whose mean-squared error dominates that of LW for Gaussian variables. Second, to further reduce the estimation error, we propose an iterative approach which approximates the clairvoyant shrinkage estimator. Convergence of this iterative method is established and a closed form expression for the limit is determined, which is referred to as the oracle approximating shrinkage (OAS) estimator. Both RBLW and OAS estimators have simple expressions and are easily implemented. Although the two methods are developed from different persepctives, their structure is identical up to specified constants. The RBLW estimator provably dominates the LW method. Numerical simulations demonstrate that the OAS approach can perform even better than RBLW, especially when n is much less than p. We also demonstrate the performance of these techniques in the context of adaptive beamforming.

Connectivity-Informed fMRI Activation Detection

“…The most well-conditioned covariance estimate of Σ is F = tr(S)/d·I d×d [12]. The idea of OAS is to shrink the ill-conditioned sample covariance, S, towards F so that a well-conditioned covariance estimate, Σ , can be obtained.…”

“…The idea of OAS is to shrink the ill-conditioned sample covariance, S, towards F so that a well-conditioned covariance estimate, Σ , can be obtained. Specifically, OAS optimizes the following cost function [12]:…”

“…Assuming brain regions displaying functional correlations at rest are more likely to co-activate during task, incorporating RS-connectivity information should improve task activation detection. To learn the connectivity structure from RS data, we employ and compare graphical LASSO (GL) [11] and the oracle approximating shrinkage (OAS) technique [12]. The resulting connectivity information is then used as a prior on the task activation effects.…”

A New Approach to Image Denoising by Patch-Based Algorithm

“…Consider the following basic image model: elements where the pixels are concatenated along a fixed lexicographic ordering. As with all patch -based techniques, the size of image patches must be described in advance [31], [32], [35]. Traditionally, the size of the image patch is a parameter-free that specifies how stochastic the user believes the image to be.…”

A Revised Inference for Correlated Topic Model

“…, r D are drawn. Chen et al [5] give the following matrix as a "naive but most well-conditioned estimate" for T :F = Tr(T ) K I, where Tr(·) is the trace of a matrix and I is the identity matrix.F is a diagonal matrix whose diagonal entries are all equal to the average of the diagonal elements ofT . Our approach usesF in place of the first term of the right hand side of Eq.…”