Multicategory Proximal Support Vector Machine Classifiers

Abstract

Abstract. Given a dataset, each element of which labeled by one of k labels, we construct by a very fast algorithm, a k-category proximal support vector machine (PSVM) classifier. Proximal support vector machines and related approaches (Fung & Mangasarian, 2001; can be interpreted as ridge regression applied to classification problems (Evgeniou, Pontil, & Poggio, 2000). Extensive computational results have shown the effectiveness of PSVM for two-class classification problems where the separating plane is constructed in time that can be as little as two orders of magnitude shorter than that of conventional support vector machines. When PSVM is applied to problems with more than two classes, the well known one-from-the-rest approach is a natural choice in order to take advantage of its fast performance. However, there is a drawback associated with this one-from-the-rest approach. The resulting two-class problems are often very unbalanced, leading in some cases to poor performance. We propose balancing the k classes and a novel Newton refinement modification to PSVM in order to deal with this problem. Computational results indicate that these two modifications preserve the speed of PSVM while often leading to significant test set improvement over a plain PSVM one-from-the-rest application. The modified approach is considerably faster than other one-from-the-rest methods that use conventional SVM formulations, while still giving comparable test set correctness.

Supervised classification and mathematical optimization

“…Among others, it is worthwhile mentioning the so-called Least Squares SVM proposed in [104,221] and the Proximal Support Vector Machine in [98,101]. In both cases, inequality (19) must be satisfied without slack, and deviation variables ξ i are now free.…”

“…In (18), they both take Ψ(t) = 1 2 t 2 and g(t) = C t 2 . However, the objective in [98,101] has an additional term, namely 1 2 β 2 . In short, both methods seek two parallel hyperplanes, each associated with one of the groups, such that the points are clustered around their corresponding hyperplane.…”

“…For more than two classes, different procedures exist to reduce the analysis to (a series of) 2-class problems, as reviewed e.g. in [101,120,123,198,230,233].…”

Dynamic class imbalance learning for incremental LPSVM

“…For LPSVM class imbalance learning, Fung et al proposed a weighted LPSVM (wLPSVM) (Fung & Mangasarian, 2005;Zhuang et al, 2005), in which the classic LPSVM optimization (7) is revised to…”

“…A serious class imbalance may even lead to the classifier completely lose its classification capability, classifying data samples all into the majority class. wLPSVM (Fung & Mangasarian, 2005;Zhuang et al, 2005) has been proposed to solve the class imbalance problem for batch LPSVM learning, in which the contribution of two classes is balanced by weights calculated on the basis of the class distribution ratio.…”