On the effect of calibration in classifier combination

Bella, Antonio; Ferri, Cèsar; Hernández-Orallo, José; Ramírez-Quintana, Marïa José

doi:10.1007/s10489-012-0388-2

Cited by 45 publications

(43 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We hope that the proposed decompositions provide deeper insight into the causes behind losses and facilitate development of better classification methods, as knowledge about calibration loss has already delivered several calibration methods, see e.g. [2]. Proof of Theorem 3: In Section 4 we proved that both methods provide adjusted scores, so we only need to prove Eq.(1).…”

Section: Discussionmentioning

confidence: 92%

“…For log-loss the decomposition for the two models is 0.717 = 0.090 + 0.628 and 0.684 = 0.056 + 0.628, respectively. In practice, calibration has proved to be an efficient way of decreasing proper scoring rule loss [2]. Calibrating a model means learning a calibration mapping from the model output scores to the respective calibrated probability scores.…”

Section: Calibrated Scores C and The Decomposition L = Cl + Rlmentioning

confidence: 99%

“…The decomposition of expected loss corresponding to any proper scoring rule into calibration loss and refinement loss has facilitated the development of calibration methods (i.e. calibration loss reduction methods) which have been shown to be beneficial for classification performance [2]. Another decomposition 1 splits refinement loss into uncertainty minus resolution [5,9].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Kull

Flach

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. Abstract. There are several reasons to evaluate a multi-class classifier on other measures than just error rate. Perhaps most importantly, there can be uncertainty about the exact context of classifier deployment, requiring the classifier to perform well with respect to a variety of contexts. This is commonly achieved by creating a scoring classifier which outputs posterior class probability estimates. Proper scoring rules are loss evaluation measures of scoring classifiers which are minimised at the true posterior probabilities. The well-known decomposition of the proper scoring rules into calibration loss and refinement loss has facilitated the development of methods to reduce these losses, thus leading to better classifiers. We propose multiple novel decompositions including one with four terms: adjustment loss, post-adjustment calibration loss, grouping loss and irreducible loss. The separation of adjustment loss from calibration loss requires extra assumptions which we prove to be satisfied for the most frequently used proper scoring rules: Brier score and log-loss. We propose algorithms to perform adjustment as a simpler alternative to calibration.

show abstract

Section: Discussionmentioning

confidence: 92%

Section: Calibrated Scores C and The Decomposition L = Cl + Rlmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Kull

Flach

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…The effect of calibration in classifier combination is studied by [2]. Perhaps closest in spirit to our work in this paper is the work by [6] who propose methods to identify and remove unreliable classifiers in a one-vs-one setting.…”

Section: Related Workmentioning

confidence: 99%

“…2 Any calibration procedure which transforms values of g(x) with a calibration map can decrease the group-wise calibration loss but not the grouping loss, which is inherent to the model. Grouping loss arises from the model's decision to group certain instances together with the same probability estimate whereas the true probabilities are different.…”

Section: Calibration and Reliabilitymentioning

confidence: 99%

Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy

Kull

Flach

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. Abstract. We propose a general method to assess the reliability of two-class probabilities in an instance-wise manner. This is relevant, for instance, for obtaining calibrated multi-class probabilities from two-class probability scores. The LS-ECOC method approaches this by performing least-squares fitting over a suitable error-correcting output code matrix, where the optimisation resolves potential conflicts in the input probabilities. While this gives all input probabilities equal weight, we would like to spend less effort fitting unreliable probability estimates. We introduce the concept of a reliability map to accompany the more conventional notion of calibration map; and LS-ECOC-R which modifies LS-ECOC to take reliability into account. We demonstrate on synthetic data that this gets us closer to the Bayes-optimal classifier, even if the base classifiers are linear and hence have high bias. Results on UCI data sets demonstrate that multi-class accuracy also improves.

show abstract

Development of Genome-Derived Tumor Type Prediction to Inform Clinical Cancer Care

et al. 2020

View full text Add to dashboard Cite

Diagnosing the site of origin for cancer is a pillar of disease classification that has directed clinical care for more than a century. Even in an era of precision oncologic practice, in which treatment is increasingly informed by the presence or absence of mutant genes responsible for cancer growth and progression, tumor origin remains a critical factor in tumor biologic characteristics and therapeutic sensitivity. OBJECTIVE To evaluate whether data derived from routine clinical DNA sequencing of tumors could complement conventional approaches to enable improved diagnostic accuracy. DESIGN, SETTING, AND PARTICIPANTS A machine learning approach was developed to predict tumor type from targeted panel DNA sequence data obtained at the point of care, incorporating both discrete molecular alterations and inferred features such as mutational signatures. This algorithm was trained on 7791 tumors representing 22 cancer types selected from a prospectively sequenced cohort of patients with advanced cancer. RESULTS The correct tumor type was predicted for 5748 of the 7791 patients (73.8%) in the training set as well as 8623 of 11 644 patients (74.1%) in an independent cohort. Predictions were assigned probabilities that reflected empirical accuracy, with 3388 cases (43.5%) representing high-confidence predictions (>95% probability). Informative molecular features and feature categories varied widely by tumor type. Genomic analysis of plasma cell-free DNA yielded accurate predictions in 45 of 60 cases (75.0%), suggesting that this approach may be applied in diverse clinical settings including as an adjunct to cancer screening. Likely tissues of origin were predicted from targeted tumor sequencing in 95 of 141 patients (67.4%) with cancers of unknown primary site. Applying this method prospectively to patients under active care enabled genome-directed reassessment of diagnosis in 2 patients initially presumed to have metastatic breast cancer, leading to the selection of more appropriate treatments, which elicited clinical responses. CONCLUSIONS AND RELEVANCE These results suggest that the application of artificial intelligence to predict tissue of origin in oncologic practice can act as a useful complement to conventional histologic review to provide integrated pathologic diagnoses, often with important therapeutic implications.

show abstract

On the effect of calibration in classifier combination

Cited by 45 publications

References 33 publications

Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy

Development of Genome-Derived Tumor Type Prediction to Inform Clinical Cancer Care

Contact Info

Product

Resources

About