Trace optimization and eigenproblems in dimension reduction methods

Abstract

Abstract. This paper gives an overview of the eigenvalue problems encountered in areas of data mining that are related to dimension reduction. Given some input high-dimensional data, the goal of dimension reduction is to map them to a lowdimensional space such that certain properties of the initial data are preserved. Optimizing the above properties among the reduced data can be typically posed as a trace optimization problem that leads to an eigenvalue problem. There is a rich variety of such problems and the goal of this paper is to unravel relations between them as well as to discuss effective solution techniques. First, we make a distinction between projective methods that determine an explicit linear projection from the high-dimensional space to the low-dimensional space, and nonlinear methods where the mapping between the two is nonlinear and implicit. Then, we show that all of the eigenvalue problems solved in the context of explicit projections can be viewed as the projected analogues of the so-called nonlinear or implicit projections. We also discuss kernels as a means of unifying both types of methods and revisit some of the equivalences between methods established in this way. Finally, we provide some illustrative examples to showcase the behavior and the particular characteristics of the various dimension reduction methods on real world data sets.

Dimensionality Reduction on SPD Manifolds: The Emergence of Geometry-Aware Methods

“…until convergence two-stage procedure. First, we fix F (W ) (i.e., assume that it does not on W ), and compute the solution of the resulting approximation of (35), which can be achieved by computing the m smallest eigenvectors of F (W ) Kokiopoulou et al (2011). Given the new W , we update F (W ), and iterate.…”

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives

“…[Benson et al, 2015, Cunningham and Ghahramani, 2015, Kokiopoulou et al, 2011, Wu et al, 2016. For example, the standard PCA, PLS, orthonormalized PLS (OPLS) and CCA can be formulated as the following EVD/GEVD problems PCA :…”

SICA: subjectively interesting component analysis

“…The solution to this problem consists of a matrix W ∈ R d×k whose k column vectors are the eigenvectors that corresponding to the top-k eigenvalues of the matrix X λ 1 |E| L + λ 2 n I n X ∈ R d×d (Kokiopoulou et al 2011). The computational complexity of finding an optimal projection W consists of two parts: (1) solving a convex optimization problem to obtain the background distribution.…”