Input space versus feature space in kernel-based methods

Abstract

This paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector (SV) kernel functions. We first discuss the geometry of feature space. In particular, we review what is known about the shape of the image of input space under the feature space map, and how this influences the capacity of SV methods. Following this, we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel, using the example of the class of inhomogeneous polynomial kernels, which are often used in SV pattern recognition. We then discuss the connection between feature space and input space by dealing with the question of how one can, given some vector in feature space, find a preimage (exact or approximate) in input space. We describe algorithms to tackle this issue, and show their utility in two applications of kernel methods. First, we use it to reduce the computational complexity of SV decision functions; second, we combine it with the Kernel PCA algorithm, thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real-world data.

Multivariate Calibration with Least-Squares Support Vector Machines

“…16,37 Finally, performing kernel-based nonlinear mapping is shown to perform well but, thus far, it is not used to retrieve physicochemical information. Reconstructing the mapping according to Schölkopf et al, 38 for example, and knowing what kind of mapping is preferred for specific features (e.g., spectral bands) can increase the knowledge of the problem. This in turn can give further directions to interpret and improve the results.…”

Maximum Margin Projection Subspace Learning for Visual Data Analysis

“…that maps (usually non-linearly) the input feature vectors x i to a possibly high dimensional space F , called feature space, which usually has the structure of a Hilbert space [32], [33], where the data are supposed to be linearly or near linearly separable. The exact form of the mapping function is not required to be known, since all required subsequent operations of the learning algorithm are expressed in terms of dot products between the input vectors in the Hilbert space performed by the kernel trick [34].…”

On the Number of Modes of a Gaussian Mixture

“…Thus, the new point x (τ +1) is the conditional mean of the mixture under the current point x (τ ) . This is formally akin to clustering by deterministic annealing (Rose, 1998), to algorithms for finding pre-images in kernel-based methods (Schölkopf et al, 1999) and to mean-shift algorithms (section 5.2).…”