Bounding the generalization error of convex combinations of classifiers: balancing the dimensionality and the margins

Abstract

A problem of bounding the generalization error of a classifier f ∈ conv(H), where H is a "base" class of functions (classifiers), is considered. This problem frequently occurs in computer learning, where efficient algorithms of combining simple classifiers into a complex one (such as boosting and bagging) have attracted a lot of attention. Using Talagrand's concentration inequalities for empirical processes, we obtain new sharper bounds on the generalization error of combined classifiers that take into account both the empirical distribution of "classification margins" and an "approximate dimension" of the classifiers and study the performance of these bounds in several experiments with learning algorithms. 1991 AMS subject classification: primary 62G05, secondary 62G20, 60F15

Complexities of convex combinations and bounding the generalization error in classification

“…The second term of the bound is of the order ( √ nδ) −1 , and will also be small for large n, which makes the bound meaningful. This result was extended by Schapire and Singer in [29] to classes of realvalued functions, namely, to so-called VC-subgraph classes (for definition see [32]), and was further extended in several directions in [19] and [21]. The main idea of this follow-up work was to replace the second term of the bound proved by Schapire et al [28] by a function ε n (F; δ; t) that has better dependence on the sample size n and on the margin parameter δ.…”

“…In [21] Koltchinskii, Panchenko and Lozano proved the bounds on generalization error under more general assumption on the entropy of the class F :…”

“…The idea of using a certain measure of complexity of individual convex combinations already appeared in [21], where the authors suggested a way to use a rate of decay of weights λ j in the convex combination f = T j=1 λ j h j to improve the bound on the generalization error of f. This measure, called approximate γ-dimension, is defined as follows. Let us assume that the weights are arranged in the decreasing order |λ 1 | ≥ |λ 2 | ≥ · · · .…”

“…, T , satisfying the conditions f = T j=1 λ j h j , T j=1 |λ j | ≤ 1 and T j=d+1 |λ j | ≤ γ. Note that in [21] the authors dealt with the symmetric convex hull, so the coefficients λ j are not necessarily positive. The γ-dimension of f will be denoted by d(f ; γ).…”

“…This is an improvement over (2.4), which can be seen by comparing the infimum over γ of the expression in the bound with the value of the expression for γ = 1 and noting that d(f ; 1) = 0. For example, if the weights decrease polynomially |λ j | ∼ j −β , β > 1, or exponentially |λ j | ∼ e −βj , β > 0, then explicit minimization over γ shows that in these cases (2.8) can be a substantial improvement over (2.4) (see examples in [21]). Our first result in this paper also deals with bounding the generalization error of a classifier f = T j=1 λ j h j ∈ F = conv(H) in terms of complexity measures taking into account the sparsity of the weights λ j .…”