Deep Linear Discriminant Analysis

Abstract

We introduce Deep Linear Discriminant Analysis (DeepLDA) which learns linearly separable latent representations in an end-to-end fashion. Classic LDA extracts features which preserve class separability and is used for dimensionality reduction for many classification problems. The central idea of this paper is to put LDA on top of a deep neural network. This can be seen as a non-linear extension of classic LDA. Instead of maximizing the likelihood of target labels for individual samples, we propose an objective function that pushes the network to produce feature distributions which: (a) have low variance within the same class and (b) high variance between different classes. Our objective is derived from the general LDA eigenvalue problem and still allows to train with stochastic gradient descent and back-propagation. For evaluation we test our approach on three different benchmark datasets (MNIST, CIFAR-10 and STL-10). DeepLDA produces competitive results on MNIST and CIFAR-10 and outperforms a network trained with categorical cross entropy (having the same architecture) on a supervised setting of STL-10.

Regularized Deep Linear Discriminant Analysis

“…To make a comparison with DLDA, the same network architecture and parameters in Dorfer et al (2015) were used in the following experiments. The network architecture renamed as DorferNet is depicted in Fig.…”

“…By focusing on directions in the latent space with smallest discriminative power, DLDA learns linearly separable hidden representations with similar discriminative power in all directions of the latent space. Although Dorfer et al (2015) showed DLDA outperforms DNN with CCE under the same network architecture, there is still some improvement potential in DLDA.…”

“…Since each eigenvalue v i is the quantitative measurement of the discrimination ability in the direction of the corresponding eigenvector u i , as a non-linear extension of the classic LDA, Deep Linear Discriminant Analysis(DLDA) replaces the original Categorical Cross Entropy(CCE) loss function with eigenvalue-based loss function to encourage deep neural network(DNN) to learn feature representations with discriminative distribution parameters (Dorfer et al (2015)). To avoid DNN focusing on maximizing the eigenvalues of which the eigenvectors already separate the classes and ignoring the eigenvalues whose eigenvectors poorly discriminate the classes, the optimization object of DNN is limited to the smallest of all c − 1 valid eigenvalues, so that all of the c − 1 latent feature dimensions would be trained on balance.…”

“…The iteration proceeds until the centroids remain the same. Hyperplanes: Dorfer et al (2015) proposed to rely on the distances to the linear decision hyperplanes (Friedman et al (2001)) for classification. e = {e i } c−1 i=1 are the corresponding eigenvectors, based on the LDA projection matrix A := [e 1 , .…”

“…Inspired by Kernel PCA, nonlinear kernel methods are also introduced to LDA (Mika et al (1999)), but those approaches are either limited by some fixed nonlinear transformations or have very complicated computation (Li et al (2019)). DNN has been widely recognized as a successful nonlinear representation learner, to utilize such talent of DNN, Dorfer et al (2015) put LDA on top of DNN to form Deep Linear Discriminant Analysis(DLDA). Instead of maximizing the likelihood of target labels as conventional CCE, the eigenvalues along the discriminant eigenvector directions are maximized.…”