Estimating covariance and precision matrices along subspaces

Abstract

We study the accuracy of estimating the covariance and the precision matrix of a D-variate sub-Gaussian distribution along a prescribed subspace or direction using the finite sample covariance. Our results show that the estimation accuracy depends almost exclusively on the components of the distribution that correspond to desired subspaces or directions. This is relevant and important for problems where the behavior of data along a lower-dimensional space is of specific interest, such as dimension reduction or structured regression problems. We also show that estimation of precision matrices is almost independent of the condition number of the covariance matrix. The presented applications include direction-sensitive eigenspace perturbation bounds, relative bounds for the smallest eigenvalue, and the estimation of the single-index model. For the latter, a new estimator, derived from the analysis, with strong theoretical guarantees and superior numerical performance is proposed.

Estimating multi-index models with response-conditional least squares

“…Our main result (Corollary 8) establishes convergence at rate N −1/2 , while additionally tracking the dependence on the hyperparameter J. Convergence will be proved using the Davis-Kahan theorem followed by concentration inequalities. For hyperparameter characterization, it will be crucial to exploit anisotropy using the techniques recently derived in [27].…”

“…A similar observation will be established for higher dimensional index spaces in (11) below. Anisotropic concentration bounds for ordinary least squares vectors have been recently provided in [27]. To restate the bounds, we introduce a directional sub-Gaussian condition number κ J, := κ(P J, , X| J, ) ∨ κ(Q J, , X| J, ), where (recall…”

“…As described in [27], κ(L, X) is related to the restricted matrix condition number defined byκ(L, Cov (X)) := L Cov (X) † L L Cov (X) L , which measures the heterogeneity of eigenvalues of Cov (X), when restricting the eigenspaces to Im(L). In fact, if X follows a normal distribution, the sub-Gaussian norm is a tight variance proxy and κ(L, X) differs fromκ(L, Cov (X)) by a constant factor that only depends on the precise definition of the sub-Gaussian norm.…”

“…Proof. The concentration bounds are precisely [27,Lemma 14] adjusted to the notation used here. (10) follows from b J, = P J, Σ † J, Cov (X| J, , Y | J, ) and [27, Lemma 6.5].…”

“…Before providing a general literature review in the next section, we note that this work builts on [27,28], which analyze estimators based on responseconditional least squares vectors for the single-index model and a nonlinear generalization thereof. While we can resort here to some results developed in [27] for determining the accuracy of local least squares vectors, the analysis requires extension to the multi-index case. Furthermore, in the present paper we derive a regression analysis of the link function, complementing index space estimation by RCLS and beyond.…”