Convexity, Classification, and Risk Bounds

Abstract

Many of the classification algorithms developed in the machine learning literature, including the support vector machine and boosting, can be viewed as minimum contrast methods that minimize a convex surrogate of the 0-1 loss function. The convexity makes these algorithms computationally efficient. The use of a surrogate, however, has statistical consequences that must be balanced against the computational virtues of convexity. To study these issues, we provide a general quantitative relationship between the risk as assessed using the 0-1 loss and the risk as assessed using any nonnegative surrogate loss function. We show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function: that it satisfy a pointwise form of Fisher consistency for classification. The relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications. We also present a refined version of this result in the case of low noise. Finally, we present applications of our results to the estimation of convergence rates in the general setting of function classes that are scaled convex hulls of a finite-dimensional base class, with a variety of commonly used loss functions.

Revisiting the Bradley-Terry model and its application to information retrieval

“…Since this empirical risk function is almost impossible to be minimized due to computational complexity, an alternative way is to use a convex surrogate loss φ of the 0-1 loss (Bartlett et al, 2006;Zhang, 2004). That is, we estimate f by minimizing the surrogated empirical risk…”

Soft-max boosting

“…This is known as the problem of calibration, which studies how small the suboptimality gap, measured in term of the surrogate risk, should be to achieve a suboptimality gap, measured in term of the risk of interest, of a given size. This has been studied in the binary cost-insensitive case (Bartlett et al 2006), in the binary cost-sensitive case (Steinwart 2007) and in the costsensitive multi-classe case (Pires et al 2013), for the convex surrogate proposed in Lee et al (2004). However, it has also been advocated recently that a surrogate loss function should be guess-averse (Beijbom et al 2014), in the sense that the loss should encourage more correct classifications than arbitrary guesses.…”

“…B Matthieu Geist matthieu.geist@supelec.fr; matthieu.geist@centralesupelec.fr 1 IMS -MaLIS Research Group and UMI 2958 (GeorgiaTech-CNRS), CentraleSupélec, Metz, France using such surrogates does not come without guarantees (Bartlett et al 2006), we propose an alternative approach in this article, focusing directly on the primary loss of interest.…”

Supervised classification and mathematical optimization

“…As we will discuss in Section 3, and already pointed out e.g. in [17], many of the optimization problems encountered fall within the area of (smooth) Convex Programming. However, other areas of Mathematical Optimization play a notable role, among others, Global Optimization [9,13,51,128,160,245], Linear Programming [94,158,205] Mixed-Integer Programming [25,39,50,77,220,228], Nonsmooth Optimization [7,13,44,51,222,223], Multicriteria and Multi-Objective Programming [68,93,181,248] and Robust Optimization [224].…”