Estimating Expected Calibration Errors

Posocco, Nicolas; Bonnefoy, Antoine

doi:10.1007/978-3-030-86380-7_12

Cited by 10 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model calibration estimates a crucial aspect of ML to ensure the predicted probabilities align with the true probabilities of events and poorly calibrated models may provide misleading confidence scores, impacting the interpretability and trustworthiness of predictions [38, 39]. However, few studies focused on evaluation of the calibration of the classification models investigated in the clinical settings [36, 38, 40]. In our study, we generated the calibration curves for the four models to investigate the relationship between the mean predicted probabilities of the positive class and the observed fraction of positive instances.…”

Section: Discussionmentioning

confidence: 99%

“…In our study, we generated the calibration curves for the four models to investigate the relationship between the mean predicted probabilities of the positive class and the observed fraction of positive instances. Additionally, we calculated the Expected Calibration Error (ECE) as a key metric used to quantify the calibration performance of a model [36]. Our ECE results revealed good calibration of the four models we investigated for prediction of acute pain intensity, MEDD, and analgesic efficacy in patients with OC/OPC receiving RT.…”

Section: Discussionmentioning

confidence: 99%

“…This curve allows for a qualitative assessment of how well the predicted probabilities align with the true outcomes. Additionally, the Expected Calibration Error (ECE) was computed for each model [36, 37]. ECE quantifies the average discrepancy between the predicted probabilities and the observed frequencies, providing a scalar measure of calibration performance.…”

Section: Methodsmentioning

confidence: 99%

“…Specifically, ECE was calculated as the mean absolute difference between the observed and predicted probabilities across predefined bins. A lower ECE value indicates better calibration [36].…”

Section: Model Evaluationmentioning

confidence: 99%

See 3 more Smart Citations

Comparison of Machine Leaning Models for Prediction of Acute Pain Severity and On-Treatment Opioid Utilization in Oral Cavity and Oropharyngeal Cancer Patients Receiving Radiation Therapy: Exploratory Analysis from a Large-Scale Retrospective Cohort

Salama,

Humbert-Vidan,

Godinich

et al. 2024

Preprint

View full text Add to dashboard Cite

BackgroundAcute pain is a common and debilitating symptom experienced by oral cavity and oropharyngeal cancer (OC/OPC) patients undergoing radiation therapy (RT). Uncontrolled pain can result in opioid overuse and increased risks of long-term opioid dependence. The specific aim of this exploratory analysis was the prediction of severe acute pain and opioid use in the acute on-treatment setting, to develop risk-stratification models for pragmatic clinical trials.Materials and MethodsA retrospective study was conducted on 900 OC/OPC patients treated with RT during 2017 to 2023. Clinical data including demographics, tumor data, pain scores and medication data were extracted from patient records. On-treatment pain intensity scores were assessed using a numeric rating scale (0-none, 10-worst) and total opioid doses were calculated using morphine equivalent daily dose (MEDD) conversion factors. Analgesics efficacy was assessed based on the combined pain intensity and the total required MEDD. ML models, including Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), and Gradient Boosting Model (GBM) were developed and validated using ten-fold cross-validation. Performance of models were evaluated using discrimination and calibration metrics. Feature importance was investigated using bootstrap and permutation techniques.ResultsFor predicting acute pain intensity, the GBM demonstrated superior area under the receiver operating curve (AUC) (0.71), recall (0.39), and F1 score (0.48). For predicting the total MEDD, LR outperformed other models in the AUC (0.67). For predicting the analgesics efficacy, SVM achieved the highest specificity (0.97), and best calibration (ECE of 0.06), while RF and GBM achieved the same highest AUC, 0.68. RF model emerged as the best calibrated model with ECE of 0.02 for pain intensity prediction and 0.05 for MEDD prediction. Baseline pain scores and vital signs demonstrated the most contributed features for the different predictive models.ConclusionThese ML models are promising in predicting end-of-treatment acute pain and opioid requirements and analgesics efficacy in OC/OPC patients undergoing RT. Baseline pain score, vital sign changes were identified as crucial predictors. Implementation of these models in clinical practice could facilitate early risk stratification and personalized pain management. Prospective multicentric studies and external validation are essential for further refinement and generalizability.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Model Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Comparison of Machine Leaning Models for Prediction of Acute Pain Severity and On-Treatment Opioid Utilization in Oral Cavity and Oropharyngeal Cancer Patients Receiving Radiation Therapy: Exploratory Analysis from a Large-Scale Retrospective Cohort

Salama,

Humbert-Vidan,

Godinich

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…They further introduce another binning-based estimator, which is also biased. Consequently, [28] concluded that current calibration error estimators are unfit for the low data regime. In later sections, we will further demonstrate that, even if there exists a perfect estimator, the TCE p and CWCE p fail to quantify the extent to which a model violates condition 2.1 of being strongly calibrated.…”

Section: Calibration Errorsmentioning

confidence: 99%

Better Uncertainty Calibration via Proper Scores for Classification and Beyond

Gruber¹,

Buettner²

2022

Preprint

View full text Add to dashboard Cite

With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on evaluating deep neural networks in terms of uncertainty calibration. Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent. In this work, we introduce the framework of proper calibration errors, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties. This upper bound allows us to reliably estimate the calibration improvement of any injective recalibration method in an unbiased manner. We demonstrate that, in contrast to our approach, the most commonly used estimators are substantially biased with respect to the true improvement of recalibration methods.

show abstract