Random projections as regularizers: learning a linear discriminant from fewer observations than dimensions

Abstract

Abstract. We prove theoretical guarantees for an averaging-ensemble of randomly projected Fisher Linear Discriminant classifiers, focusing on the case when there are fewer training observations than data dimensions. The specific form and simplicity of this ensemble permits a direct and much more detailed analysis than existing generic tools in previous works. In particular, we are able to derive the exact form of the generalization error of our ensemble, conditional on the training set, and based on this we give theoretical guarantees which directly link the performance of the ensemble to that of the corresponding linear discriminant learned in the full data space. To the best of our knowledge these are the first theoretical results to prove such an explicit link for any classifier and classifier ensemble pair. Furthermore we show that the randomly projected ensemble is equivalent to implementing a sophisticated regularization scheme to the linear discriminant learned in the original data space and this prevents overfitting in conditions of small sample size where pseudo-inverse FLD learned in the data space is provably poor. Our ensemble is learned from a set of randomly projected representations of the original high dimensional data and therefore for this approach data can be collected, stored and processed in such a compressed form. We confirm our theoretical findings with experiments, and demonstrate the utility of our approach on several datasets from the bioinformatics domain and one very high dimensional dataset from the drug discovery domain, both settings in which fewer observations than dimensions are the norm. A preliminary version of this work received the best paper award at the 5th Asian Conference on Machine Learning.

Correlation-based linear discriminant classification for gene expression data

“…It uses all variables and correlations among variables in subspaces composed of randomly selected variables. The analysis was conducted in Matlab according to the algorithm described in Durrant and Kabán (2015). The same parameters as in Durrant and Kabán (2015) were also used.…”

“…The analysis was conducted in Matlab according to the algorithm described in Durrant and Kabán (2015). The same parameters as in Durrant and Kabán (2015) were also used. The dimension of the subspaces, k, was set to (N -2) / 2 to optimize the classification, where N was the number of samples in the training dataset.…”

“…Cross-validation (CV) (Durrant and Kabán, 2015;Tan et al, 2015) was used to evaluate the performance of each classifier using the simulated datasets and real datasets. In the process of CV, the dataset was split into testing set, of which the classes were assumed unknown, and training set, by which the classifier was trained.…”

“…For simulated datasets, the number of samples was small and leave-one-out CV was used. For real datasets, CV was performed according to Durrant and Kabán (2015), in which we ran experiments on 100 independent splits, and in each split we took 12 samples for testing and used the remainder for training. MRs were averaged over the 100 splits.…”

“…Some of these methods do not consider multivariate correlations (i.e., they are not correlation-based), e.g., the diagonal linear discriminant analysis (DLDA), which assumes that the within-class covariance matrices are diagonal (Slawski et al, 2008). Other methods account for correlations (i.e., they are correlation-based), e.g., the ensemble of random subspace (RS) Fisher linear discriminant (FLD) classifier, which averages the decisions of the base FLD classifiers in lower-dimensional subspaces and gene-gene correlations might be utilized (Durrant and Kabán, 2015).…”