Calibration of Machine Learning Models

Bella, Antonio; Ferri, Cèsar; Hernández-Orallo, José; Ramírez-Quintana, Marïa José

doi:10.4018/978-1-60566-766-9.ch006

Cited by 47 publications

(40 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, the score yielded by the above methods is not calibrated for comparison across domains. Calibration [18] is an important desired property of machine learning systems, in particular when multiple models are operating in conjunction [19]. As we propose training modularized re-rankers, calibration becomes a critical requirement for selecting top hypotheses across all the domains.…”

Section: Introductionmentioning

confidence: 99%

A Re-Ranker Scheme For Integrating Large Scale NLU Models

Gupta

Ananthakrishnan

et al. 2018

2018 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Large scale Natural Language Understanding (NLU) systems are typically trained on large quantities of data, requiring a fast and scalable training strategy. A typical design for NLU systems consists of domain-level NLU modules (domain classifier, intent classifier and named entity recognizer). Hypotheses (NLU interpretations consisting of various intent+slot combinations) from these domain specific modules are typically aggregated with another downstream component. The re-ranker integrates outputs from domain-level recognizers, returning a scored list of cross domain hypotheses. An ideal reranker will exhibit the following two properties: (a) it should prefer the most relevant hypothesis for the given input as the top hypothesis and, (b) the interpretation scores corresponding to each hypothesis produced by the re-ranker should be calibrated. Calibration allows the final NLU interpretation score to be comparable across domains. We propose a novel re-ranker strategy that addresses these aspects, while also maintaining domain specific modularity. We design optimization loss functions for such a modularized re-ranker and present results on decreasing the top hypothesis error rate as well as maintaining the model calibration. We also experiment with an extension involving training the domain specific re-rankers on datasets curated independently by each domain to allow further asynchronization.

show abstract

Section: Introductionmentioning

confidence: 99%

A Re-Ranker Scheme For Integrating Large Scale NLU Models

Gupta

Ananthakrishnan

et al. 2018

2018 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

show abstract

“…A single approach is binning, which just uses a sliding window over the estimated valueŷ. This approach resembles binning calibration in classification [Bella et al 2009] More formally, the BIN method is defined as follows:…”

Section: Ncde Through Enrichment Methodsmentioning

confidence: 99%

“…In fact, we envisage an intensive work in this line, similar to what was done in the last decade for probability estimation in classification, where many classification methods were rethought and redesigned to get good probabilities or good rankings (e.g., probability estimation trees [Provost and Domingos 2003;Ferri et al 2003;Ferri et al 2002]). Similarly, an important progress was made in calibration methods [Zadrozny and Elkan 2002;Bella et al 2009]. This parallel between enrichment and calibration could be exploited to find theoretical foundations and connections with some of the methods inspired or closely related to calibration methods in classification, such as BIN or uKNC, or to refine them.…”

Section: Alternative Approaches and Other Applicationsmentioning

confidence: 99%

“…This suggests that the view off in terms of calibration may be more clarifying. In classification, we say that a model is calibrated [Cohen and Goldszmidt 2004;Bella et al 2009] if the following property holds: given an estimated probabilityPr(c|x) = p for class c then the true class is c in a proportion of p cases. In order to apply this definition to deterministic functions, bins of equal or similar estimated probability are used to derive the proportion.…”

Section: The True and Estimated Conditional Densities Reliability Anmentioning

confidence: 99%

“…Some of the techniques and notions for addressing these cases are cost matrices, cost-sensitive classification [Elkan 2001], ROC analysis [Swets et al 2000;Fawcett 2006a], threshold-choice methods [Hernández-Orallo et al 2012;, calibration [Cohen and Goldszmidt 2004;Bella et al 2009] and, of course, the notions of soft classifiers (outputting a score or probability) versus the notion of crisp classifiers (just outputting a label). Certainly, there have also been a few efforts to find the parallel of these techniques for regression.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Probabilistic Reframing for Cost-Sensitive Regression

́ndez-Orallo

2014

ACM Trans. Knowl. Discov. Data

View full text Add to dashboard Cite

Common day applications of predictive models usually involve a full use of the available contextual information. When the operating context changes, one may fine-tune the by-default (incontextual) prediction or may even abstain from predicting a value (a reject). Global reframing solutions, where the same function is applied to adapt the estimated outputs to a new cost context, is a possible solution here. An alternative approach, which has not been studied in a comprehensive way for regression in the knowledge discovery and data mining literature, is the use of a local (e.g., probabilistic) reframing approach, where decisions are made according to the estimated output and a reliability, confidence or probability estimation. In this paper, we advocate for a simple two-parameter (mean and variance) approach, working with a normal conditional probability density. Given the conditional mean produced by any regression technique, we develop lightweight 'enrichment' methods that produce good estimates of the conditional variance, which are used by the probabilistic (local) reframing methods. We apply these methods to some very common families of cost-sensitive problems, such as optimal predictions in (auction) bids, asymmetric loss scenarios and rejection rules.

show abstract

Missing the missing values: The ugly duckling of fairness in machine learning

et al. 2021

View full text Add to dashboard Cite

Nowadays, there is an increasing concern in machine learning about the causes underlying unfair decision making, that is, algorithmic decisions discriminating some groups over others, especially with groups that are defined over protected attributes, such as gender, race and nationality. Missing values are one frequent manifestation of all these latent causes: protected groups are more reluctant to give information that could be used against them, sensitive information for some groups can be erased by human operators, or data acquisition may simply be less complete and systematic for minority groups. However, most recent techniques, libraries and experimental results dealing with fairness in machine learning have simply ignored missing data. In this paper, we present the first comprehensive analysis of the relation between missing values and algorithmic fairness for machine learning: (1) we analyse the sources of missing data and bias, mapping the common causes, (2) we find that rows containing missing values are usually fairer than the rest, which should discourage the consideration of missing values as the uncomfortable ugly data that different techniques and libraries for handling algorithmic bias get rid of at the first occasion, (3) we studyThis is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Calibration of Machine Learning Models

Cited by 47 publications

References 3 publications

A Re-Ranker Scheme For Integrating Large Scale NLU Models

A Re-Ranker Scheme For Integrating Large Scale NLU Models

Probabilistic Reframing for Cost-Sensitive Regression

Missing the missing values: The ugly duckling of fairness in machine learning

Contact Info

Product

Resources

About