Training sparse least squares support vector machines by the QR decomposition

Abstract

The solution of an LS-SVM has suffered from the problem of non-sparseness. The paper proposed to apply the KMP algorithm, with the number of support vectors as the regularization parameter, to tackle the non-sparseness problem of LS-SVMs. The idea of the kernel matching pursuit (KMP) algorithm was first revisited from the perspective of the QR decomposition of the kernel matrix on the training set. Strategies are further developed to select those support vectors which minimize the leave-one-out cross validation error of the resultant sparse LS-SVM model. It is demonstrated that the LOOCV of the sparse LS-SVM can be computed accurately and efficiently. Experimental results on benchmark datasets showed that, compared to the SVM and variants sparse LS-SVM models, the proposed sparse LS-SVM models developed upon KMP algorithms maintained comparable performance in terms of both accuracy and sparsity.

An accelerated sequential minimal optimization method for the least squares support vector machine

“…The improved algorithm only computes the conjugate gradient once. In addition, Xia used QR decomposition to train sparse LS-SVM models similar to SVM [11]. Sparse kernel matrix is easy to calculate and fast in training.…”

Sparse support vector machines with L0 approximation for ultra-high dimensional omics data

“…It is formulated into a convex quadratic programming and can be solved effciently. However, standard SVM, minimizing the hinge loss function and L 2 norm, only leads to sparsity for the dual variables, but not the primal variables [20,21,22]. To handle the big omics data problem with a lot of features, support vector machines with other penalties including L 1 and elastic net have been proposed for feature selection and prediction [23,24,25,26,27,28].…”