Misclassification Probability Bounds for Multivariate Gaussian Classes

2014

Mach Learn

Abstract. We prove theoretical guarantees for an averaging-ensemble of randomly projected Fisher Linear Discriminant classifiers, focusing on the case when there are fewer training observations than data dimensions. The specific form and simplicity of this ensemble permits a direct and much more detailed analysis than existing generic tools in previous works. In particular, we are able to derive the exact form of the generalization error of our ensemble, conditional on the training set, and based on this we give theoretical guarantees which directly link the performance of the ensemble to that of the corresponding linear discriminant learned in the full data space. To the best of our knowledge these are the first theoretical results to prove such an explicit link for any classifier and classifier ensemble pair. Furthermore we show that the randomly projected ensemble is equivalent to implementing a sophisticated regularization scheme to the linear discriminant learned in the original data space and this prevents overfitting in conditions of small sample size where pseudo-inverse FLD learned in the data space is provably poor. Our ensemble is learned from a set of randomly projected representations of the original high dimensional data and therefore for this approach data can be collected, stored and processed in such a compressed form. We confirm our theoretical findings with experiments, and demonstrate the utility of our approach on several datasets from the bioinformatics domain and one very high dimensional dataset from the drug discovery domain, both settings in which fewer observations than dimensions are the norm. A preliminary version of this work received the best paper award at the 5th Asian Conference on Machine Learning.

“…The proof of this lemma is similar in spirit to the one for a single FLD in [40]. For completeness we give it below.…”

Section: Generalization Error Of the Ensemble For A Fixed Training Setmentioning

confidence: 85%

Section: Proof Of Lemma 43mentioning

confidence: 99%

Random projections as regularizers: learning a linear discriminant from fewer observations than dimensions

2014

Mach Learn

“…Such bounds on the classification error for FLD in the data space are already known, for example they are given in [4,15], but in neither of these papers is classification error in the projected domain considered; indeed in [9] it is stated that establishing the probability of error for a classifier in the projected domain is, in general, a difficult problem.…”

Section: Introductionmentioning

confidence: 99%

Compressed fisher linear discriminant analysis

Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

2010

We consider random projections in conjunction with classification, specifically the analysis of Fisher's Linear Discriminant (FLD) classifier in randomly projected data spaces.Unlike previous analyses of other classifiers in this setting, we avoid the unnatural effects that arise when one insists that all pairwise distances are approximately preserved under projection. We impose no sparsity or underlying lowdimensional structure constraints on the data; we instead take advantage of the class structure inherent in the problem. We obtain a reasonably tight upper bound on the estimated misclassification error on average over the random choice of the projection, which, in contrast to early distance preserving approaches, tightens in a natural way as the number of training examples increases. It follows that, for good generalisation of FLD, the required projection dimension grows logarithmically with the number of classes. We also show that the error contribution of a covariance misspecification is always no worse in the low-dimensional space than in the initial high-dimensional space. We contrast our findings to previous related work, and discuss our insights.

“…Such bounds on the classification error for FLD in the data space are already known, for example those in [2,9], but in neither of these papers is classification error in the projected domain considered; indeed in [7] it is stated that establishing the probability of error for a classifier in the projected domain is, in general, a difficult problem.…”

Section: Introductionmentioning

confidence: 99%

A Bound on the Performance of LDA in Randomly Projected Data Spaces

2010 20th International Conference on Pattern Recognition

2010

We consider the problem of classification in nonadaptive dimensionality reduction. Specifically, we bound the increase in classification error of Fisher's Linear Discriminant classifier resulting from randomly projecting the high dimensional data into a lower dimensional space and both learning the classifier and performing the classification in the projected space. Our bound is reasonably tight, and unlike existing bounds on learning from randomly projected data, it becomes tighter as the quantity of training data increases without requiring any sparsity structure from the data.