Feature selection in SVM via polyhedral k-norm

Abstract

We treat the Feature Selection problem in the Support Vector Machine (SVM) framework by adopting an optimization model based on use of the 0 pseudo-norm. The objective is to control the number of non-zero components of normal vector to the separating hyperplane, while maintaining satisfactory classification accuracy. In our model the polyhedral norm. [k] , intermediate between. 1 and. ∞, plays a significant role, allowing us to come out with a DC (Difference of Convex) optimization problem that is tackled by means of DCA algorithm. The results of several numerical experiments on benchmark classification datasets are reported.

Polytopal balls arising in optimization

“…Let us recall that ρ d,k is introduced in [9] as an intersection of half-spaces of R d . We recover this description as a consequence of Theorem 2.1.…”

“…In particular, it is shown in [18] that these norms are a solution to an optimization problem regarding the conditional value at risk. The same norms also appear in optimization problems over sets of matrices [21], where they are called vector k-norms, and in sparse optimization [9,10].…”

“…Another approach is to replace x 0 by the difference of two of the norms introduced above. Indeed, it is observed in [9,10] that the sparsity constraint x 0 ≤ k is equivalent to the equality x (k) − x (l) = 0 for any l such that k < l ≤ d. Hence, these norms defined by (1) allow for a computationally effective way to estimate sparsity. As we mentioned above, the balls for these norms are polytopes.…”

“…when k ranges from 1 to d. It is shown in [9] that ρ d,k is the unit ball for the norm defined by (1). Note that these polytopes interpolate between the hypercube and its dual cross-polytope.…”

Proximal Operator and Optimality Conditions for Ramp Loss SVM

“…Support vector machines (SVM) were first introduced by Vapnik and Cortes [6] and have been widely applied into many fields, including text and image classification [12,25], disease detection [8,16], etc. The decision hyperplane of SVM classifier, w, x + b = 0 with w ∈ R n and b ∈ R, is trained from data set {(x i , y i ), i ∈ N m } where x i ∈ R n , y i ∈ {−1, 1} and N m := {1, 2, • • • , m} by optimizing the following problem min w∈R n ,b∈R…”