Analysis of boosting algorithms using the smooth margin function

Abstract

We introduce a useful tool for analyzing boosting algorithms called the ``smooth margin function,'' a differentiable approximation of the usual margin for boosting algorithms. We present two boosting algorithms based on this smooth margin, ``coordinate ascent boosting'' and ``approximate coordinate ascent boosting,'' which are similar to Freund and Schapire's AdaBoost algorithm and Breiman's arc-gv algorithm. We give convergence rates to the maximum margin solution for both of our algorithms and for arc-gv. We then study AdaBoost's convergence properties using the smooth margin function. We precisely bound the margin attained by AdaBoost when the edges of the weak classifiers fall within a specified range. This shows that a previous bound proved by R\"{a}tsch and Warmuth is exactly tight. Furthermore, we use the smooth margin to capture explicit properties of AdaBoost in cases where cyclic behavior occurs.Comment: Published in at http://dx.doi.org/10.1214/009053607000000785 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

On the equivalence of weak learnability and linear separability: new relaxations and efficient boosting algorithms

“…In this case, the algorithm shares the simplicity of the popular AdaBoost approach. The rate of convergence we obtain matches the rate of the AdaBoost ⋆ described by Ratsch and Warmuth [RW05] and is better than the rate obtained in Rudin et al [RSD07]. We note also that if A is γ-separable and we set ǫ = γ/2 then we would find a solution with half the optimal margin in O(log(m)/γ 2 ) iterations.…”

“…Therefore, our algorithm attains the same rate of convergence of AdaBoost while both algorithms obtain a margin which is half of the optimal margin. (See also the margin analysis of AdaBoost described in Rudin et al [RSD07]. )…”

“…However, AdaBoost does not converge to the max margin solution [RW05,RSD07]. Interestingly, the equivalence between weak learnability and linear separability is not only qualitative but also quantitative: weak learnability with edge γ is equivalent to linear separability with an ℓ 1 margin of γ.…”

Precise statements of convergence for AdaBoost and arc-gv

“…Note that α arc t is non-negative since µ arc t ≤ ρ ≤ r arc t . We start our calculation from when the smooth margin is positive; if the data is separable, one can always use AdaBoost until the smooth margin is positive (see [15]). We denote by1 the first iteration where G is positive, so g arc 1 > 0.…”

“…The smooth margin obeys a useful recursion relation which helps greatly in the analyses for both algorithms. For more detailed analysis of the smooth margin function see [14], and for detailed proofs, see the extended version of this work [15].…”