Linear discriminant analysis: A detailed tutorial

Abstract

Linear Discriminant Analysis (LDA) is a very common technique for dimensionality reduction problems as a preprocessing step for machine learning and pattern classification applications. At the same time, it is usually used as a black box, but (sometimes) not well understood. The aim of this paper is to build a solid intuition for what is LDA, and how LDA works, thus enabling readers of all levels be able to get a better understanding of the LDA and to know how to apply this technique in different applications. The paper first gave the basic definitions and steps of how LDA technique works supported with visual explanations of these steps. Moreover, the two methods of computing the LDA space, i.e. class-dependent and class-independent methods, were explained in details. Then, in a step-by-step approach, two numerical examples are demonstrated to show how the LDA space can be calculated in case of the class-dependent and class-independent methods. Furthermore, two of the most common LDA problems (i.e. Small Sample Size (SSS) and non-linearity problems) were highlighted and illustrated, and state-of-the-art solutions to these problems were investigated and explained. Finally, a number of experiments was conducted with different datasets to (1) investigate the effect of the eigenvectors that used in the LDA space on the robustness of the extracted feature for the classification accuracy, and (2) to show when the SSS problem occurs and how it can be addressed.

Deep generative variational autoencoding for replay spoof detection in automatic speaker verification

“…A known problem with GMMs, however, is that use of highdimensional (short-term) features often leads to numerical problems due to singular covariance matrices. Even if off-the-shelf dimensionality reduction methods such as principal component analysis (PCA) [18] or linear discriminant analysis (LDA) [19] prior to GMM modeling may help, they are not jointly optimized with the GMM. Is there an alternative way to learn a generative model that can handle high-dimensional inputs natively?…”

Applying Deep Neural Network (DNN) for Robust Indoor Localization in Multi-Building Environment

“…As discussed in [29,30], high dimensionality has problems, such as requiring a large amount of time, high space complexity, and high over-fitting problems. Through appropriate application of dimensionality reduction techniques, it is possible to project a set of high-dimensional vector samples into much lower dimensionality while preserving the relevant global structure information of the data [31].…”

“…LDA is fast training approach for dimension reduction, so that it helps to improve computational complexity [31]. According to [30,31], there are two types of LDA technique: class-dependent and class-independent. In class-dependent types, the lower dimensional space is calculated for each class to project its data on it.…”

“…In class representations, each class will be considered as a distinct class alongside the other classes. We used LDA as a preprocessor stage, as it is more appropriate to our datasets and helps to optimize by maximizing the between-class separations, and minimizing within-class variability [29,30]. In the LDA technique, we have calculated the mean of each class, the total mean of the datasets, the between-class matrix, and the within-class matrix step by step, and then we constructed the lower dimensional spaces as a final step.…”

“…This also makes it easy to visualize and manipulate the intrinsic structure of the projected samples. According to [30], LDA is used for supervised approaches of dimensional reductions. The main goal of the dimensionality reduction techniques is to reduce the redundant and dependent features by transforming the higher dimensional feature space, which may lead to the curse of dimensionality problem, to a space with lower dimensions.…”