Learning Eigenfunctions Links Spectral Embedding and Kernel PCA

Abstract

In this paper, we show a direct relation between spectral embedding methods and kernel PCA, and how both are special cases of a more general learning problem, that of learning the principal eigenfunctions of an operator defined from a kernel and the unknown data generating density.

Diffusion Maps for dimensionality reduction and visualization of meteorological data

“…The Nyström formula [15] is a general method to approximate the eigenfunctions ψ j (x) of a kernel from the eigenvectors ψ j (x i ) of a sample-based kernel matrix. As we shall see, in the case of DM, the formula enables to approximately compute the embedding of new patterns without computing again the eigenvalues and eigenvectors of the similarity matrix of the training sample.…”

“…, x n }, let { j }, {v j } be the eigenvalues and eigenvectors of the samplerestricted kernel matrix k ij = k(x i , x j ). Then, the general Nyström method [15] enables us to approximate the u l eigenfunctions by the following expression that extends the matrix eigenvectors v j to a new y as…”

“…This extension is the second important challenge in DM and other manifold learning methods and we will address it in this work through two different algorithms that extend the diffusion coordinates for new out-of-sample points. The first one is an extension to the non-symmetric transition matrix P of the classical Nyström formula [15] for symmetric, positive semidefinite matrices derived from a kernel that extends the eigenvectors of the sample kernel matrix to the eigenfunctions of the underlying integral operator. The second one, the Laplacian Pyramids algorithm [16], also relies on a kernel representation but starts from the discrete sample values f (x i ) (in our case, the eigenvectors of the sample based Markov transition matrix P ) of a certain function f (in our case, the general eigenfunctions), and seeks a multiscale representation of f that allows to approximate the values f (x) from an appropriate multiscale combination of the sample values f (x i ).…”

Kernel Principal Components Are Maximum Entropy Projections

“…This connection provides a metric multidimensional scaling algorithm to solve Kernel PCA instead of a eigendecomposition of the Gram matrix. Bengio et al [5] pointed out the link between Kernel PCA and spectral embedding. The direct relation resides in a more general learning problem: learning the principal eigenfunctions of operators defined from a kernel and the unknown data-generating density function.…”

Large-Scale Clustering through Functional Embedding

“…Moreover, a vector f i has to be learnt in order to embed an example x i into a K-dimensional space; each example is embedded separately, and after training, given a new example x * there is no straightforward way to embed it. This is referred to as the "out-of-sample" problem, which several authors have tried to address with special techniques (e.g., [6,16,26]). Learning a function-based embedding might prove useful to solve the above problems.…”