On order determination by predictor augmentation

Abstract

In many dimension reduction problems in statistics and machine learning, such as principal component analysis, canonical correlation analysis, independent component analysis, and sufficient dimension reduction, it is important to determine the dimension of the reduced predictor, which often amounts to estimating the rank of a matrix. This problem is called order determination. In this paper, we propose a novel and highly effective order-determination method based on the idea of predictor augmentation. We show that, if we augment the predictor by an artificially generated random vector, then the part of the eigenvectors of the matrix induced by the augmentation display a pattern that reveals information about the order to be determined. This information, when combined with the information provided by the eigenvalues of the matrix, greatly enhances the accuracy of order determination.

Dimension Estimation in Two-Dimensional PCA

“…This idea is formalized in the following paragraphs. For more details on the procedure in general, see [10] where the augmentation estimator was first proposed (in the context of vector-valued data).…”

“…In the vector setting, [10] used the median of the eigenvalues of the sample covariance matrix as an estimate for σ 2 , under the assumption that at least half of the components are noise. A similar approach can be applied in our setting: Let σ2…”

“…, p 1 , but with caution as very negative values of σ2 i − σ2 can indicate that the noise variance σ 2 is not properly estimated, which can happen, e.g., if assumption (4) does not hold. The normalization adjustment in the eigenvalue function Φ n is done, as in the bootstrap ladle estimator [12], to ensure robustness with respect to scaling of the data, whereas the constant 1 in the denominator is used for stabilization in the extreme case of noise only, that is, when d 1 = 0 [10]. The constant 1 also enhances the decreasing pattern of the eigenvalue function, especially in settings with small sample sizes where the (d + 1)st eigenvalue might not be very small.…”

“…The four models considered in the simulation study are: (10,10,3), and, similarly, Then, for each of the four models, 1000 data sets of size n = 1000 were created and the dimensions d 1 and d 2 were estimated using all possible combinations of s ∈ {1, 10, 50} and r ∈ {1, 5, 10}. To estimate σ 2 , the following three approaches were used (i) mean rule: the mean of the 50% smallest values of Ŝ, (ii) median rule: the median of Ŝ and (iii) last rule: the minimum of Ŝ.…”

“…Hence, Figure 2 on its own is not sufficient to solve the problem and, to supplement it, we propose combining it with the information contained in the eigenvectors produced by (2D) 2 PCA. This procedure, proposed originally in [10] for vector-valued data under the name of "predictor augmentation", aims to create a "reverse scree plot" where the curve stays flat until the optimal cutoff point is reached and increases afterward. The sum of the "reverse scree plot" and the scree plot is then minimized at the optimal dimension, enabling its straightforward detection, both visually and automatically.…”