On the Equivalence of the SMO and MDM Algorithms for SVM Training

Abstract

Abstract. SVM training is usually discussed under two different algorithmic points of view. The first one is provided by decomposition methods such as SMO and SVMLight while the second one encompasses geometric methods that try to solve a Nearest Point Problem (NPP), the GilbertSchlesinger-Kozinec (GSK) and Mitchell-Demyanov-Malozemov (MDM) algorithms being the most representative ones. In this work we will show that, indeed, both approaches are essentially coincident. More precisely, we will show that a slight modification of SMO in which at each iteration both updating multipliers correspond to patterns in the same class solves NPP and, moreover, that this modification coincides with an extended MDM algorithm. Besides this, we also propose a new way to apply the MDM algorithm for NPP problems over reduced convex hulls.

First and Second Order SMO Algorithms for LS-SVM Classifiers

“…These MVP and Second Order rules can be shown to be very intuitively derived under a point of view based on the dual gain [15]. We will follow the lines of this last work, adapting them to the dual problem (8).…”

“…As was the case for (14), in standard SVMs the selections (15) and (16) are somewhat more complex, since it must be taken into account that the coefficients are box-constrained [15]. Selection (16), by (11) or its relaxed version (12), corresponds to choosing the MVP.…”

“…The so-called Second Order selection was proposed in [10] as a better approximation than (16) to the optimal choice (15). Note that, in First Order, the cost of the search of (16) is O (1) kernel operations, if we assume that the values F i =W ·˜ (X i ) − y i are cached throughout the algorithm.…”

Coresets for polytope distance

“…A variant of Gilbert's algorithm, the GJK algorithm, is a popular algorithm for collision detection in 3 dimensional space [12]. Another important variant of this, called the MDM algorithm [20], is in fact equivalent to one of the most used SVM training algorithms, SMO [23,17]. For SVM training, [16] obtained good experimental results with a combination of Gilbert's and the MDM algorithm.…”

A method to improve support vector machine based on distance to hyperplane

“…In this method, second order approximation information of the dual function is used to search the second index so that the number of iterations can be decreased greatly. The second order rules have been derived based on the dual gain [10] .…”