META-DES: A dynamic ensemble selection framework using meta-learning

Cruz, Rafael M. O.; Sabourin, Robert; Cavalcanti, George D. C.; Ren, Tsang Ing

doi:10.1016/j.patcog.2014.12.003

Cited by 211 publications

(142 citation statements)

References 38 publications

Supporting

Mentioning

140

Contrasting

Unclassified

Order By: Relevance

“…Finally, a series of methods have been developed recently under the common name 'dynamic model selection' [10], [16], [17], [17]- [21]. These approaches take an ensemble of base classifiers (e.g, from bagging), then attempt to learn a high-level classification model using, for example, instanceinstance similarities and model-model correlations, as input features.…”

Section: Ice Differs From Most Existing Ensemble Methods Significantlmentioning

confidence: 99%

A Novel Multiple Classifier Generation and Combination Framework Based on Fuzzy Clustering and Individualized Ensemble Construction

Zand

Ruan

2019

2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA)

View full text Add to dashboard Cite

Multiple classifier system (MCS) has become a successful alternative for improving classification performance. However, studies have shown inconsistent results for different MCSs, and it is often difficult to predict which MCS algorithm works the best on a particular problem. We believe that the two crucial steps of MCS -base classifier generation and multiple classifier combination, need to be designed coordinately to produce robust results. In this work, we show that for different testing instances, better classifiers may be trained from different subdomains of training instances including, for example, neighboring instances of the testing instance, or even instances far away from the testing instance. To utilize this intuition, we propose Individualized Classifier Ensemble (ICE). ICE groups training data into overlapping clusters, builds a classifier for each cluster, and then associates each training instance to the top-performing models while taking into account model types and frequency. In testing, ICE finds the k most similar training instances for a testing instance, then predicts class label of the testing instance by averaging the prediction from models associated with these training instances. Evaluation results on 49 benchmarks show that ICE has a stable improvement on a significant proportion of datasets over existing MCS methods. ICE provides a novel choice of utilizing internal patterns among instances to improve classification, and can be easily combined with various classification models and applied to many application domains.

show abstract

Section: Ice Differs From Most Existing Ensemble Methods Significantlmentioning

confidence: 99%

A Novel Multiple Classifier Generation and Combination Framework Based on Fuzzy Clustering and Individualized Ensemble Construction

Zand

Ruan

2019

2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA)

View full text Add to dashboard Cite

show abstract

“…A total of 15 sets of meta-features are considered, with ten sets proposed in this paper, and five coming from our previous work [6]. Each set f i captures a different property of the behavior of the base classifier, and can be seen as a different criterion to dynamically estimate the level of competence of the base classifier, such as the classification performance estimated in a local region of the feature space and the classifier confidence for the classification of the input sample.…”

Section: Meta-feature Extractionmentioning

confidence: 99%

“…Then, only the classifier(s) that attain a certain competence level, are used to predict the label of the given test sample. Recent works in the MCS literature have shown that dynamic ensemble selection (DES) techniques achieve higher classification accuracy when compared to static ones [5,6,7]. This is especially true for ill-defined problems, i.e., for problems where the size of the training data is small, and there are not enough data available to train the classifiers [8,9].…”

Section: Introductionmentioning

confidence: 99%

“…Several criteria have previously been proposed, based on different sources of information, such as the classifier local accuracy estimates in small regions of the feature space surrounding the query instance, called the region of competence [11,7], probabilistic models [12,13,14], ranking [15] and classifier behavior [16,8]. In our previous work [6], we proposed a novel DES framework using meta-learning, called META-DES.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection

2017

Self Cite

View full text Add to dashboard Cite

Dynamic ensemble selection (DES) techniques work by estimating the competence level of each classifier from a pool of classifiers, and selecting only the most competent ones for the classification of a specific test sample. The key issue in DES is defining a suitable criterion for calculating the classifiers' competence. There are several criteria available to measure the level of competence of base classifiers, such as local accuracy estimates and ranking. However, using only one criterion may lead to a poor estimation of the classifier's competence. In order to deal with this issue, we have proposed a novel dynamic ensemble selection framework using meta-learning, called META-DES. A meta-classifier is trained, based on the meta-features extracted from the training data, to estimate the level of competence of a classifier for the classification of a given query sample. An important aspect of the META-DES framework is that multiple criteria can be embedded in the system encoded as different sets of meta-features. However, some DES criteria are not suitable for every classification problem. For instance, local accuracy estimates may produce poor results when there is a high degree of overlap between the classes. Moreover, a higher classification accuracy can be obtained if the performance of the meta-classifier is optimized for the corresponding data. In this paper, we propose a novel version of the META-DES framework based on the formal definition of the Oracle, called META-DES.Oracle. The Oracle is an abstract method that represents an ideal classifier selection scheme. A meta-feature selection scheme using an overfitting cautious Binary Particle Swarm Optimization (BPSO) is proposed for improving the performance of the meta-classifier. The difference between the outputs obtained by the meta-classifier and those presented by the Oracle is minimized. Thus, the meta-classifier is expected to obtain results that are similar to the Oracle. Experiments carried out using 30 classifi-1 arXiv:1811.00217v1 [cs.LG] 1 Nov 2018 cation problems demonstrate that the optimization procedure based on the Oracle definition leads to a significant improvement in classification accuracy when compared to previous versions of the META-DES framework and other state-of-the-art DES techniques.

show abstract

“…Some works of dynamic classifier selection (DCS) [7], [8] use other measures, beyond accuracy, to calculate the competence. Recent works on DCS [9], [10], [11] use the composition of many measures to determine the competence of the classifiers, selecting and combining them to predict the class of the test pattern.…”

Section: Introductionmentioning

confidence: 99%

Evaluating Competence Measures for Dynamic Regressor Selection

Moura

Cavalcanti

Oliveira

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Dynamic regressor selection (DRS) systems work by selecting the most competent regressors from an ensemble to estimate the target value of a given test pattern. This competence is usually quantified using the performance of the regressors in local regions of the feature space around the test pattern. However, choosing the best measure to calculate the level of competence correctly is not straightforward. The literature of dynamic classifier selection presents a wide variety of competence measures, which cannot be used or adapted for DRS. In this paper, we review eight measures used with regression problems, and adapt them to test the performance of the DRS algorithms found in the literature. Such measures are extracted from a local region of the feature space around the test pattern, called region of competence, therefore competence measures. To better compare the competence measures, we perform a set of comprehensive experiments of 15 regression datasets. Three DRS systems were compared against individual regressor and static systems that use the Mean and the Median to combine the outputs of the regressors from the ensemble. The DRS systems were assessed varying the competence measures. Our results show that DRS systems outperform individual regressors and static systems but the choice of the competence measure is problem-dependent.

show abstract

META-DES: A dynamic ensemble selection framework using meta-learning

Cited by 211 publications

References 38 publications

A Novel Multiple Classifier Generation and Combination Framework Based on Fuzzy Clustering and Individualized Ensemble Construction

A Novel Multiple Classifier Generation and Combination Framework Based on Fuzzy Clustering and Individualized Ensemble Construction

META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection

Evaluating Competence Measures for Dynamic Regressor Selection

Contact Info

Product

Resources

About