An Evaluation of Grading Classifiers

Seewald, Alexander K.; Fürnkranz, Johannes

doi:10.1007/3-540-44816-0_12

Cited by 85 publications

(45 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We then summarize the results of several recent studies in stacking [8,11,12,10,13]. Motivated by these, we introduce a modified stacking approach based on classification via linear regression [11].…”

Section: Stackingmentioning

confidence: 99%

See 1 more Smart Citation

Stacking with an Extended Set of Meta-level Attributes and MLR

Ženko

Džeroski

2002

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We propose a new set of meta-level features to be used for learning how to combine classifier predictions with stacking. This set includes the probability distributions predicted by the base-level classifiers and a combination of these with the certainty of the predictions. We use these features in conjunction with multi-response linear regression (MLR) at the meta-level. We empirically evaluate the proposed approach in comparison to several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking. Our approach performs better than existing stacking approaches and also better than selecting the best classifier from the ensemble by cross validation (unlike existing stacking approaches, which at best perform comparably to it).

show abstract

Section: Stackingmentioning

confidence: 99%

“…Seewald and Fürnkranz [10] propose a method for combining classifiers called grading that learns a meta-level classifier for each base-level classifier. The metalevel classifier predicts whether the base-level classifier is to be trusted (i.e., whether its prediction will be correct).…”

Section: Recent Advancesmentioning

confidence: 99%

Stacking with an Extended Set of Meta-level Attributes and MLR

Ženko

Džeroski

2002

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In the Grading method as proposed by [6], when there is a tie in the likelihood of an instance belonging to different classes, the meta-classifier checks which of the class has higher prior probability and accordingly makes a decision. We suggest an alternative scheme in which instead of completely ignoring the predictions of some of the base classifiers (the classifiers with higher probability of being wrong than correct), the grader should assign a delta (close to zero) probability to all such classifiers being correct.…”

Section: Tie Breaking For Gradingmentioning

confidence: 99%

“…Grading. The defining feature of methods in this category (also known as referee method [5,6]) is that, instead of directly finding the relationship between the predictions of the base classifier and the actual class (as in stacking); the meta-classifier grades the base-classifiers, and selects either a single or subset of base-classifier(s) which are likely to be correct for the given test instance. The intuition behind grading is that in large datasets where there may be multiple functions defining the relationship between predictor and response variables, it is important to choose the correct function for any given test instance.…”

Section: Introductionmentioning

confidence: 99%

Error-Sensitive Grading for Model Combination

Singhi

Liu

2005

Machine Learning: ECML 2005

View full text Add to dashboard Cite

Abstract. Ensemble learning is a powerful learning approach that combines multiple classifiers to improve prediction accuracy. An important decision while using an ensemble of classifiers is to decide upon a way of combining the prediction of its base classifiers. In this paper, we introduce a novel grading-based algorithm for model combination, which uses cost-sensitive learning in building a meta-learner. This method distinguishes between the grading error of classifying an incorrect prediction as correct, and the other-way-round, and tries to assign appropriate costs to the two types of error in order to improve performance. We study issues in error-sensitive grading, and then with extensive experiments show the empirically effectiveness of this new method in comparison with representative meta-classification techniques.

show abstract

“…In contrast, delegation produces models which are completely and exclusively defined in terms of the original attributes and class. Arbitrating (Ortega et al, 2001) and grading (Seewald & Fürnkranz, 2001) are also related to delegation, but both learn external referees to assess the probability of error of each classifier from the pool of base classifiers, and their areas of expertise. No new attributes are generated.…”

Section: Introductionmentioning

confidence: 99%

Delegating classifiers

Ferri

Flach

Hernández-Orallo

2004

Twenty-First International Conference on Machine Learning - ICML '04

View full text Add to dashboard Cite

A sensible use of classifiers must be based on the estimated reliability of their predictions. A cautious classifier would delegate the difficult or uncertain predictions to other, possibly more specialised, classifiers. In this paper we analyse and develop this idea of delegating classifiers in a systematic way. First, we design a two-step scenario where a first classifier chooses which examples to classify and delegates the difficult examples to train a second classifier. Secondly, we present an iterated scenario involving an arbitrary number of chained classifiers. We compare these scenarios to classical ensemble methods, such as bagging and boosting. We show experimentally that our approach is not far behind these methods in terms of accuracy, but with several advantages: (i) improved efficiency, since each classifier learns from fewer examples than the previous one; (ii) improved comprehensibility, since each classification derives from a single classifier; and (iii) the possibility to simplify the overall multiclassifier by removing the parts that lead to delegation.

show abstract

An Evaluation of Grading Classifiers

Cited by 85 publications

References 9 publications

Stacking with an Extended Set of Meta-level Attributes and MLR

Stacking with an Extended Set of Meta-level Attributes and MLR

Error-Sensitive Grading for Model Combination

Delegating classifiers

Contact Info

Product

Resources

About