An eigenanalysis of data centering in machine learning

Abstract

Many pattern recognition methods rely on statistical information from centered data, with the eigenanalysis of an empirical central moment, such as the covariance matrix in principal component analysis (PCA), as well as partial least squares regression, canonical-correlation analysis and Fisher discriminant analysis. Recently, many researchers advocate working on non-centered data. This is the case for instance with the singular value decomposition approach, with the (kernel) entropy component analysis, with the information-theoretic learning framework, and even with nonnegative matrix factorization. Moreover, one can also consider a noncentered PCA by using the second-order non-central moment.The main purpose of this paper is to bridge the gap between these two viewpoints in designing machine learning methods. To provide a study at the cornerstone of kernel-based machines, we conduct an eigenanalysis of the inner product matrices from centered and noncentered data. We derive several results connecting their eigenvalues and their eigenvectors. Furthermore, we explore the outer product matrices, by providing several results connecting the largest eigenvectors of the covariance matrix and its non-centered counterpart. These results lay the groundwork to several extensions beyond conventional centering, with the weighted mean shift, the rank-one update, and the multidimensional scaling. Experiments conducted on simulated and real data illustrate the relevance of this work.

A Regression Perspective on Generalized Distance Covariance and the Hilbert–Schmidt Independence Criterion

“…However, assuming w.l.o.g. that p 0 ≥ • • • ≥ p m , it follows directly from the interlacing property in [27], Theorem 3, that, for j = 0, . .…”

DIAS: A Data-Informed Active Subspace Regularization Framework for Inverse Problems

“…From a statistical perspective, the uncentered covariance matrix of the form (1) is not common, though it has been investigated. Comparisons between centered and uncentered covariance matrices have been carried out for general covariance matrices in the context of principal component analysis [31][32][33][34]. We now address the difference between uncentered and centered covariance matrices in the context of active subspaces.…”