Online training on a budget of support vector machines using twin prototypes

Abstract

This paper proposes twin prototype support vector machine (TVM), a constant space and sublinear time support vector machine (SVM) algorithm for online learning. TVM achieves its favorable scaling by memorizing only a fixed-size data summary in the form of example prototypes and their associated information during training. In addition, TVM guarantees that the optimal SVM solution is maintained on all prototypes at any time. To maximize the accuracy of TVM, prototypes are constructed to approximate the data distribution near the decision boundary. Given a new training example, TVM is updated in three steps. First, the new example is added as a new prototype if it is near the decision boundary. If this happens, to maintain the budget, either the prototype farthest away from the decision boundary is removed or two near prototypes are selected and merged into a single one. Finally, TVM is updated by incremental and decremental techniques to account for the change. Several methods for prototype merging were proposed and experimentally evaluated. TVM algorithms with hinge loss and ramp loss were implemented and thoroughly tested on 12 large datasets. In most cases, the accuracy of low-budget TVMs was comparable with the resource-unconstrained SVMs. Additionally, the TVM accuracy was substantially larger than that of SVM trained on a random sample of the same size. Even larger difference in accuracy was observed when comparing with Forgetron, a popular budgeted kernel perceptron algorithm. As expected, the difference in accuracy between hinge loss and ramp loss TVM was negligible and hinge loss version is preferable due to its lower computational cost. The results illustrate that highly accurate online SVMs could be trained from arbitrary large data streams using devices with severely limited memory budgets.

MEEM: Robust Tracking via Multiple Experts Using Entropy Minimization

“…The base tracker for our multi-expert framework is inspired by the online SVM algorithm of [26], which makes use of prototype sets to gain an improved approximation to the offline SVM. For our base tracker, the algorithm of [26] is reformulated to better suit the tracking problem.…”

“…For our base tracker, the algorithm of [26] is reformulated to better suit the tracking problem. Note that the following formulation is for a stand-alone tracker, which is independent of the multi-expert framework.…”

“…Since positive samples usually have much lower diversity than negative ones in the tracking problem, the algorithm of [26] tends to make the positive prototype instances collapse into a single instance. To avoid this for our SVM tracker, positive prototype instances are not merged until their number |Q + | reaches a predefined bound B + .…”

“…To implement the base tracker in our multi-expert framework, we adopt an online SVM algorithm [26] that approximates the offline version by employing compact prototype sets, which summarize the effects of all previous training samples near the decision boundary and thereby avoids hard pruning of training samples, as needed in [31,21,11]. However, in [26] the algorithm is not evaluated in the tracking problem, so we carefully reformulated it to account for specific characteristics of the tracking problem.…”

“…However, in [26] the algorithm is not evaluated in the tracking problem, so we carefully reformulated it to account for specific characteristics of the tracking problem. Furthermore, we use a linear kernel and the feature mapping technique of [19] to efficiently find nonlinear decision boundaries in the original feature space.…”