Induced Subgraph Game for Ensemble Selection

Abstract

Ensemble methodology has proved to be one of the strongest machine learning techniques. In spite of its huge success, most ensemble methods tend to generate unnecessarily large number of classifiers, which entails an increase in memory storage, computational cost, and even a reduction in the generalization performance of the ensemble. Ensemble selection addresses these shortcomings by searching for a fraction of individual classifiers that performs as good as, or better than the entire ensemble. In this paper, we formulate ensemble selection problem as a coalitional game played on a graph. The proposed game aims at capturing two crucial concepts that affect the performance of an ensemble: accuracy and diversity. Most importantly, it ranks every classifier based on its contribution in keeping a proper balance between these two notions using Shapley value. To demonstrate the validity and the effectiveness of the proposed approach, we carried out experimental comparisons with some major selection techniques based on 35 UCI benchmark datasets. The results reveal that our approach significantly improves the original ensemble and performs better than the other methods in terms of classification accuracy, pruning ratio, and computational cost.

Shapley Values as a Strategy for Ensemble Weights Estimation

“…This approach evaluated the contribution of every available classifier by incorporating ensemble characteristics such as individual accuracies and group variety into a Shapley value. The difference in approach in [32] from ours is that we choose not to measure ensemble diversity explicitly.…”

“…Shapley values [18] were also applied in the ranking-based ensemble selection approach [32] that was based on the inducted subgraph game from combinatorics. This approach evaluated the contribution of every available classifier by incorporating ensemble characteristics such as individual accuracies and group variety into a Shapley value.…”

“…Data characteristics and dataset sources are described in Table 2. The Bank Marketing dataset was also used for experiments in [32]. Preparation of the Bank Marketing dataset consisted of transforming targeted value from "Yes" and "No" to 1 and 0, respectively, and one hot encoding of categorical data.…”

An efficient ensemble pruning approach based on simple coalitional games

“…Then, they ordered the ensemble members according to their contributions in descending order. In the same regard, Ykhlef and Bouchaffra formulated ensemble pruning problem as an induced subgraph game [17] . Their approach first ranks every classifier by considering the ensemble diversity and the individual accuracies based on Shapley value; then, it constitutes the pruned ensemble by aggregating the top N members.…”

“…Then, it ranks every individual learner based on Banzhaf power index (line [11][12][13]). Finally, it sets the pruned ensemble as the minimal winning coalition made of the best ranked learners (line [14][15][16][17][18]). More specifically, the algorithm iteratively chooses, from among the classifiers not yet selected, the classifier with the highest rank, and adds it to the selected set ω until ω wins.…”