Kernel Optimization in Discriminant Analysis

Abstract

Kernel mapping is one of the most used approaches to intrinsically derive nonlinear classifiers. The idea is to use a kernel function which maps the original nonlinearly separable problem to a space of intrinsically larger dimensionality where the classes are linearly separable. A major problem in the design of kernel methods is to find the kernel parameters that make the problem linear in the mapped representation. This paper derives the first criterion that specifically aims to find a kernel representation where the Bayes classifier becomes linear. We illustrate how this result can be successfully applied in several kernel discriminant analysis algorithms. Experimental results using a large number of databases and classifiers demonstrate the utility of the proposed approach. The paper also shows (theoretically and experimentally) that a kernel version of Subclass Discriminant Analysis yields the highest recognition rates.

Low-Rank and Eigenface Based Sparse Representation for Face Recognition

“…Recent years, the kernel optimization based algorithms are popular and competitive. It is necessary to compare our algorithm with some kernel optimization based algorithms mentioned in [36] on the AR database. For simplification, we also resized the images into 29×21 pixels.…”

“…The images in session 1 were used for training and those in session 2 for testing. One could refer to [36] for details. The results are shown in Table 4 (some results originate from [36] directly).…”

Multiobjective Optimization for Model Selection in Kernel Methods in Regression

“…However, to achieve this, the newly derived objective function needs to be combined with the classical one measuring its fitness (i.e., how well the function estimates the sample vectors). Classical solutions would be to use the sum or product of the two objective functions [47]. However, we have shown that these solutions do not generally yield desirable results in kernel methods in regression.…”

“…The most pressing question for us is to show that this derived solution yields lower prediction errors than simpler, more straight forward approaches. Two such criteria are the sum and product of the two terms to be minimized [47], given by $$Q^{sum}\left(bold\theta \right)=E_f\left(bold\theta \right)+\nu E_c\left(bold\theta \right).$$ and $$Q^{pro}\left(bold\theta \right)=E_f\left(bold\theta \right)E_c{\left(bold\theta \right)}^{\gamma },$$ where ν and γ are regularization parameters needed to be selected. Note that minimizing (30) is equivalent to minimizing $$\lg Q^{pro}\left(bold\theta \right)=\lg E_f\left(bold\theta \right)+\gamma \lg E_c\left(bold\theta \right).$$ which is the logarithm of (30).…”

“…The selection of the appropriate parameters of a kernel is a challenging one [47]. If the parameters were chosen to minimize the model fit, we would generally have an over-fitting to the training data.…”