Learning a hyperplane classifier by minimizing an exact bound on the VC dimension1

Abstract

The VC dimension measures the complexity of a learning machine, and a low VC dimension leads to good generalization. While SVMs produce state-of-the-art learning performance, it is well known that the VC dimension of a SVM can be unbounded; despite good results in practice, there is no guarantee of good generalization. In this paper, we show how to learn a hyperplane classifier by minimizing an exact, or Θ bound on its VC dimension. The proposed approach, termed as the Minimal Complexity Machine (MCM), involves solving a simple linear programming problem. Experimental results show, that on a number of benchmark datasets, the proposed approach learns classifiers with error rates much less than conventional SVMs, while often using fewer support vectors. On many benchmark datasets, the number of support vectors is less than one-tenth the number used by SVMs, indicating that the MCM does indeed learn simpler representations.

Learning a Fuzzy Hyperplane Fat Margin Classifier with Minimum VC dimension

“…Table 1 summarizes information about the number of samples and features of each dataset. [5] for the sake of brevity; an added reason is that a fair comparison would be between two methods that use a fuzzy methodology.…”

“…However, according to Burges [4], SVMs can have a very large VC dimension, and that "at present there exists no theory which shows that good generalization performance is guaranteed for SVMs". In recent work [5], we have shown how to learn a bounded margin hyperplane classifier, termed as the Minimal Complexity Machine (MCM) by minimizing an exact bound on its VC dimension.…”

Benchmarking KAZE and MCM for Multiclass Classification

“…In contrast, MCM guarantees good generalization accuracy by obtaining a better bound (lower and upper) over the VC dimension while also achieving excellent training error rates. In addition as noted in [12], the number of support vectors obtained by MCM are comparatively less than that of SVM. This makes MCM suitable for complex classification tasks while also providing opportunity to reduce the overall overhead during classification.…”

“…For the other key operations in object classification, Support Vector Machines(SVM) [6] have been the traditional choice. Recently, Minimal Complexity Machine (MCM) [12] has shown to outperform SVM in terms accuracy, computational complexity as well as sparse representation of the features. The strongest argument in favor of MCM is its provably good generalization accuracy and requirement of far less number of support vectors as compared to SVMs.…”

Feature Selection through Minimization of the VC dimension

“…The notion of an exact bound means that the objective being minimized bounds the VC dimension from both above and below; this means that the two are close to each other. The theory that allows us to do so is motivated by the recently proposed Minimal Complexity Machine (MCM) [14,15]. The MCM shows that it is possible to learn a hyperplane classifier by minimizing an exact bound on the VC dimension.…”