Classification model selection via bilevel programming

Abstract

Support vector machines and related classification models require the solution of convex optimization problems that have one or more regularization hyper-parameters. Typically, the hyper-parameters are selected to minimize the cross-validated estimates of the out-of-sample classification error of the model. This cross-validation optimization problem can be formulated as a bilevel program in which the outer-level objective minimizes the average number of misclassified points across the cross-validation folds, subject to inner-level constraints such that the classification functions for each fold are (exactly or nearly) optimal for the selected hyper-parameters. Feature selection is included in the bilevel program in the form of bound constraints in the weights. The resulting bilevel problem is converted to a mathematical program with linear equilibrium constraints, which is solved using state-of-the-art optimization methods. This approach is significantly more versatile than commonly used grid search procedures, enabling, in particular, the use of models with many hyper-parameters. Numerical results demonstrate the practicality of this approach for model selection in machine learning.

Modeling Choices for Virtual Screening Hit Identification

“…Recent efforts [30][31][32] have focused on a bilevel optimization approach to implementing K-fold cross-validation. This approach may be beneficial to learn VISHID models by permitting faster SVM model Figure 11.…”

Bilevel model selection for support vector machines

“…Many of the algorithms presented in this monograph have similar structures. It has already been demonstrated that bilevel machine learning problems perform fairly well with regards to generalization error [3,36] when solved using SQP-based methods such as filter. It should be noted that since the ultimate goal is to produce good generalization, the solutions found need not necessarily be highly accurate or global optimal.…”

“…In this section, we extend the parameter selection idea introduced in the previous section to support vector classification and show how the bilevel formulation can handle a large number of hyper-parameters; this is a review of work that was first introduced in [36]. The inner-level problem is the standard SVC model [12] augmented with additional feature selection constraints.…”

“…The scalars, b t , are collected into the vector b ∈ R T . Using this additional notation, the formulation of cross validation for SV classification as a bilevel program [36] is shown below.…”

“…Inexact cross validation was used to solve bilevel cross validation for support vector regression, (2.5), in [3] and support vector classification, (4.1), in [36] using filter. In spite of the fact that filter provides no guarantee of global optimality and generally converges to locally optimal solutions, this method performed well with regard to generalization error, indicating that local optimal solutions can be practically satisfactory.…”