Precise statements of convergence for AdaBoost and arc-gv

We introduce a useful tool for analyzing boosting algorithms called the ``smooth margin function,'' a differentiable approximation of the usual margin for boosting algorithms. We present two boosting algorithms based on this smooth margin, ``coordinate ascent boosting'' and ``approximate coordinate ascent boosting,'' which are similar to Freund and Schapire's AdaBoost algorithm and Breiman's arc-gv algorithm. We give convergence rates to the maximum margin solution for both of our algorithms and for arc-gv. We then study AdaBoost's convergence properties using the smooth margin function. We precisely bound the margin attained by AdaBoost when the edges of the weak classifiers fall within a specified range. This shows that a previous bound proved by R\"{a}tsch and Warmuth is exactly tight. Furthermore, we use the smooth margin to capture explicit properties of AdaBoost in cases where cyclic behavior occurs.Comment: Published in at http://dx.doi.org/10.1214/009053607000000785 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

show abstract

“…Sections 8, 9 and 10 contain proofs from Sections 3, 5, 6 and 7. Preliminary and less detailed statements of these results appear in [25,26].…”

Section: Convergence Properties Of New and Old Algorithmsmentioning

confidence: 94%

Analysis of boosting algorithms using the smooth margin function

Rudin

Schapire²,

Daubechies

2007

Ann. Statist.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The adversarial multi-armed bandit problem can be treated within the class of Exponentially Weighted Average Forecaster algorithms [17]. Typically these algorithms maintain a probability distribution over the arms and draws a random arm from this distribution at each step.…”

Section: Exp4: Combination Of Expertsmentioning

confidence: 99%

“…We first specify a reward function for each information source. Let = ∑ ℎ = ~ ℎ be the edge [17] of the base hypothesis ℎ chosen by the base learner at time step $t$. Here the edge helps define reward functions in the proposed algorithm.…”

Section: Eboost: Combining Adaboost and Exp4mentioning

confidence: 99%

Combining Expert Strategies in Multimodal Classification

Peng

2014

AMM

View full text Add to dashboard Cite

In multimodal classification, we look for a set of strategies for mining and exploiting the most informative modalities for a given situation. These strategies are computations performed by the algorithms. In this paper, we propose to consider strategies as advice given to an algorithm by “expert.” There can be several classification strategies. Each strategy makes different assumptions regarding the fidelity of a sensor modality and uses different data to arrive at its estimates. Each strategy may place different trust in a sensor at different times, and each may be better in different situations. In this paper, we introduce a novel algorithm for combining expert strategies to achieve robust classification performance in a multimodal setting. We provide experimental results using real world examples to demonstrate the efficacy of the proposed algorithm.

show abstract

“…But it still has its downsides, as Breiman's quote above indicates. There is evidence both for and against the power of the margin theory to predict the quality of the generalization performance [Breiman, 1999, Rudin et al, 2004, 2007a,b, Reyzin and Schapire, 2006. But the most striking problem is that the margin bound is very loose: It does not explain the precise behavior of the error.…”

Section: The Margin Theorymentioning

confidence: 99%

On the Convergence Properties of Optimal AdaBoost

Belanich¹,

Ortiz²

2012

Preprint

View full text Add to dashboard Cite

AdaBoost is one of the most popular machine-learning algorithms. It is simple to implement and often found very effective by practitioners, while still being mathematically elegant and theoretically sound. AdaBoost's behavior in practice, and in particular the test-error behavior, has puzzled many eminent researchers for over a decade: It seems to defy our general intuition in machine learning regarding the fundamental trade-off between model complexity and generalization performance. In this paper, we establish the convergence of "Optimal AdaBoost," a term coined by Rudin, Daubechies, and Schapire in 2004. We prove the convergence, with the number of rounds, of the classifier itself, its generalization error, and its resulting margins for fixed data sets, under certain reasonable conditions. More generally, we prove that the time/per-round average of almost any function of the example weights converges. Our approach is to frame AdaBoost as a dynamical system, to provide sufficient conditions for the existence of an invariant measure, and to employ tools from ergodic theory. Unlike previous work, we do not assume AdaBoost cycles; actually, we present empirical evidence against it on real-world datasets. Our main theoretical results hold under a weaker condition. We show sufficient empirical evidence that Optimal AdaBoost always met the condition on every real-world dataset we tried. Our results formally ground future convergencerate analyses, and may even provide opportunities for slight algorithmic modifications to optimize the generalization ability of AdaBoost classifiers, thus reducing a practitioner's burden of deciding how long to run the algorithm.

show abstract

Precise statements of convergence for AdaBoost and arc-gv

Cited by 8 publications

References 12 publications

Analysis of boosting algorithms using the smooth margin function

Analysis of boosting algorithms using the smooth margin function

Combining Expert Strategies in Multimodal Classification

On the Convergence Properties of Optimal AdaBoost

Contact Info

Product

Resources

About