Invariance of principal components under low-dimensional random projection of the data

Abstract

Algorithms that can efficiently recover principal components of high-dimensional data from compressive sensing measurements (e.g. low-dimensional random projections) of it have been an important topic of recent interest in the literature. In this paper, we show that, under certain conditions, normal principal component analysis (PCA) on such low-dimensional random projections of data actually returns the same result as PCA on the original data set would. In particular, as the number of data samples increases, the center of the randomly projected data converges to the true center of the original data (up to a known scaling factor) and the principal components converge to the true principal components of the original data as well, even if the dimension of each random subspace used is very low. Indeed, experimental results verify that this approach does estimate the original center and principal components very well for both synthetic and real-world datasets, including hyperspectral data. Its performance is even superior to that of other algorithms recently developed in the literature for this purpose.Index Terms-Compressive sensing, Principal component analysis, Random projections, Low-rank matrix recovery, Hyperspectral data 1. INTRODUCTION Principal component analysis (PCA) [1] selects the best lowdimensional linear projection of a set of data points to minimize error between the original and projected data. It can also be thought of as finding the linear subspace that maximally preserves the variance of, or in some cases, the information in, the data. PCA is frequently used for dimensionality reduction, or as a summary of interesting features of the data. It is also often used as a precursor to signal classification.To obtain the principal components (PCs) of data, one typically centers the data and then computes the eigenvectors of the data's covariance matrix, using full knowledge of all data. However, in this paper, we will show that when the PCA algorithm is instead applied to low-dimensional random projections of each data point, as are acquired in many compressive sensing (CS) measurement schemes [2], it will often return the same center (up to a known scaling factor) and principal components as it would for the original dataset.More precisely, we show that the center of the low-dimensional random projections of the data converges to the true center of the original data (up to a known scaling factor) almost surely as the number of data samples increases. We then show that under certain conditions the top d eigenvectors of the randomly projected data's covariance matrix converge to the true d principal components of the original data as the number of data samples increases. Moreover, both of the above conclusions are true regardless of how few dimensions we use in our random projections (i.e. how few CS measurements we take of each sample).Furthermore, experimentally, we find that on both synthetic and real-world examples, including hyperspectral data, normal PCA on low-dimensional random projections of the...

Random projections in reducing the dimensionality of climate simulation data

“…Results are in line with the findings of Qi and Hughes (2012), where it is theoretically verified that, although RP disperses the energy of a PC in different directions, the original PC remains as the direction with the most energy. Due to this, oscillations with similar variance can be assigned to different, adjacent components, leading to some ambiguity in the indices.…”

“…With a subspace of 10% of the original dimensions, we were able to recover the PCs explaining 96% of the variance in the original data set and with 1% we still could recover the PCs explaining 94% of the original variance. The findings of this work are supported by the results presented in Qi and Hughes (2012). In their paper, it is theoretically and experimentally shown that a normal PCA performed on low-dimensional random subspaces recovers the principal components of the original data set very well, and as the number of data samples n increases the principal components of the random subspace converge to the true original components.…”

“…This paper studies random projection (RP) as a dimensionality reduction method. It has been successfully applied in image processing (Bingham and Mannila, 2001;Goel et al, 2005;Qi and Hughes, 2012) and for text data (Bingham and Mannila, 2001). RPs fall into the theory of compressive sampling (CS), which has emerged as a novel paradigm in data sampling after the publications of Cande`s et al (2006) and Donoho (2006).…”

Sparse Sensor Placement Optimization for Classification

“…Sparse approximation applied to semantic hierarchies [41] has been shown to be efficient in categorizing between large numbers of classes [35]. Ideas from compressed sensing have also been applied to dimensionality reduction tools to develop a theory of sketched SVD based on randomly projected data [23,47,29]. Methods to promote sparsity and parsimonious representations of high-dimensional data from few measurements include the development of sparse PCA (SPCA) [64] and, in a related line of work in feature selection for classification, penalized and sparse LDA [60,16].…”

Harmonic Analysis Algorithm Based On SVD Filter

“…It is convenient to use Parks-McClellan optimal digital filter an applicable narrow and steep transitional band-pass filter [3]. The filter designed by this algorithm, however, has a higher order, implying that the execution on hardware is complex, which, as a result, will lead to a higher power consumption of the hardware system.…”