Aggregation using input–output trade-off

Abstract

In this paper, we introduce a new learning strategy based on a seminal idea of Mojirsheibani (1999, 2000, 2002a, 2002b), who proposed a smart method for combining several classifiers, relying on a consensus notion. In many aggregation methods, the prediction for a new observation x is computed by building a linear or convex combination over a collection of basic estimators r1(x), . . . , rm(x) previously calibrated using a training data set. Mojirsheibani proposes to compute the prediction associated to a new observation by combining selected outputs of the training examples. The output of a training example is selected if some kind of consensus is observed: the predictions computed for the training example with the different machines have to be "similar" to the prediction for the new observation. This approach has been recently extended to the context of regression in Biau et al. (2016).In the original scheme, the agreement condition is actually required to hold for all individual estimators, which appears inadequate if there is one bad initial estimator. In practice, a few disagreements are allowed ; for establishing the theoretical results, the proportion of estimators satisfying the condition is required to tend to 1.In this paper, we propose an alternative procedure, mixing the previous consensus ideas on the predictions with the Euclidean distance computed between entries. This may be seen as an alternative approach allowing to reduce the effect of a possibly bad estimator in the initial list, using a constraint on the inputs.We prove the consistency of our strategy in classification and in regression. We also provide some numerical experiments on simulated and real data to illustrate the benefits of this new aggregation method. On the whole, our practical study shows that our method may perform much better than the original combination technique, and, in particular, exhibit far less variance. We also show on simulated examples that this procedure mixing inputs and outputs is still robust to high dimensional inputs.

KFC: A clusterwise supervised learning procedure based on the aggregation of distances

“…An alternative definition of combined estimator suggests mixing the consensus idea with information about distances between inputs (Fischer and Mougeot (2019)). This is a way to limit the influence, if any, of a bad estimator; using at the same time information on the geometry of the inputs.…”

“…In the second step, for each divergence, a very simple predictive model is fit per cluster. The final step provides an adaptive global predictive model by aggregating, thanks to a consensus idea introduced by Mojirsheibani (1999), several models built for the different instances, corresponding to the different Bregman divergences (see also Mojirsheibani (2000); Balakrishnan and Mojirsheibani (2015); Biau et al (2016); Fischer and Mougeot (2019)). We name this procedure the KFC procedure for K-means/Fit/Consensus.…”

Kernel-Based Ensemble Learning in Python

“…In a work parallel to ours, the idea of using the distance between points in the output space is also explored by Fischer and Mougeot [8], where weights are assigned to points based on proximity of the prediction in the output space and the training data. However, the method employed (which we will now refer to as MixCobra) also uses the input data while constructing the aggregate.…”

Gradient COBRA: A kernel-based consensual aggregation for regression

“…This method has also been applied in filtering to improve the image denoising (see Guedj and Rengot (2020)). Moreover, consensual aggregation methods such as Biau et al (2016), Fischer and Mougeot (2019) and the present method are also incorporated in a three-step methodology called KFC procedure, which combines unsupervised clustering and supervised prediction for (energy) data modeling (see Has et al (2021)). Such an idea of consensual aggregation was also used in unsupervised classification known as Clustering Aggregation (see, for example, Gionis et al (2005) and Wu et al (2012)).…”