Scaling multi-class support vector machines using inter-class confusion

Abstract

Support vector machines (SVMs) excel at two-class discriminative learning problems. They often outperform generative classifiers, especially those that use inaccurate generative models, such as the naive Bayes (NB) classifier. On the other hand, generative classifiers have no trouble in handling an arbitrary number of classes efficiently, and NB classifiers train much faster than SVMs owing to their extreme simplicity. In contrast, SVMs handle multi-class problems by learning redundant yes/no (one-vs-others) classifiers for each class, further worsening the performance gap. We propose a new technique for multi-way classification which exploits the accuracy of SVMs and the speed of NB classifiers. We first use a NB classifier to quickly compute a confusion matrix, which is used to reduce the number and complexity of the two-class SVMs that are built in the second stage. During testing, we first get the prediction of a NB classifier and use that to selectively apply only a subset of the two-class SVMs. On standard benchmarks, our algorithm is 3 to 6 times faster than SVMs and yet matches or even exceeds their accuracy.

A Robust Text Classifier Based on Denoising Deep Neural Network in the Analysis of Big Data

“…The application of the earliest machine learning method is naive Bayes (NB) [3,4]. Subsequently, almost all the important machine learning algorithms have been applied to the field of text classification, for example, nearest neighbor (KNN), neural network (NN), support vector machine (SVM), decision tree, kernel learning, and some others [5][6][7][8][9][10]. SVM uses the shallow linear model to separate the objective.…”

Efficient multi-way text categorization via generalized discriminant analysis

“…Godbole et al [14] propose a new multi-class classification technique that exploits the accuracy of SVMs and the speed of Naive Bayes. It uses a Naive Bayes classifier to compute a confusion matrix quickly.…”

“…This categorization problem is usually viewed as supervised learning, where the gaol is to assign predefined category labels to unlabeled documents based on the likelihood inferred from the training set of labeled documents. Numerous approaches have been applied, including Bayesian probabilistic approaches [20,31], nearest neighbor [22,19], neural networks [33], decision trees [2], inductive rule learning [4,9], support vector machines [18,14], Maximum Entropy [26], boosting [28], and linear discriminate projection [3] (see [34] for comparative studies of text categorization methods).…”

Label-Driven Learning Framework: Towards More Accurate Bayesian Network Classifiers through Discrimination of High-Confidence Labels

“…It has been widely applied in many domains of pattern recognition, such as character recognition [10], age estimation [11] and face recognition [12]. The multistage classification methodology proposed in [13] aims at reducing mutual misclassifications among easily confused labels and consists of two stages. Firstly, an inexpensive classifier, which can discriminate among all labels, coarsely classifies a testing instance.…”