Efficient Sparse Approximation of Support Vector Machines Solving a Kernel Lasso

Abstract

Performing predictions using a non-linear support vector machine (SVM) can be too expensive in some large-scale scenarios. In the non-linear case, the complexity of storing and using the classifier is determined by the number of support vectors, which is often a significant fraction of the training data. This is a major limitation in applications where the model needs to be evaluated many times to accomplish a task, such as those arising in computer vision and web search ranking.We propose an efficient algorithm to compute sparse approximations of a non-linear SVM, i.e., to reduce the number of support vectors in the model. The algorithm is based on the solution of a Lasso problem in the feature space induced by the kernel. Importantly, this formulation does not require access to the entire training set, can be solved very efficiently and involves significantly less parameter tuning than alternative approaches. We present experiments on well-known datasets to demonstrate our claims and make our implementation publicly available.

Nonlinear Kernel Support Vector Machine with 0-1 Soft Margin Loss

“…The quantity of SVs not only determines the memory space of the learned predictor, but also the computational cost of using it. A vast body of literatures have devoted to the task to improve the sparse performance of SVM, such as [44], [45], [46], [47], [48]. Luckily, the binary classifier SVM with 0-1 loss has the innate ability to reduce the sample candidates to be SVs because for the geometrical and numerical evidence shown in this article.…”