Cal-Net: Jointly Learning Classification and Calibration On Imbalanced Binary Classification Tasks

Datta, Arghya; Flynn, Noah R.; Swamidass, S. Joshua

doi:10.1109/ijcnn52387.2021.9534411

Cited by 8 publications

(13 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With this in mind, we decided to present models that made categorical predictions. Research to reduce the negative interaction between oversampling techniques and calibration error is ongoing and future iteration of these models may be capable of reliably producing continuous risk estimates 55 . Though our models are highly sensitive, there were false positives.…”

Section: Discussionmentioning

confidence: 99%

Predicting 180-day mortality for women with ovarian cancer using machine learning and patient-reported outcome data

Sidey‐Gibbons

Sun

Schneider

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Contrary to national guidelines, women with ovarian cancer often receive treatment at the end of life, potentially due to the difficulty in accurately estimating prognosis. We trained machine learning algorithms to guide prognosis by predicting 180-day mortality for women with ovarian cancer using patient-reported outcomes (PRO) data. We collected data from a single academic cancer institution in the United States. Women completed biopsychosocial PRO measures every 90 days. We randomly partitioned our dataset into training and testing samples. We used synthetic minority oversampling to reduce class imbalance in the training dataset. We fitted training data to six machine learning algorithms and combined their classifications on the testing dataset into an unweighted voting ensemble. We assessed each algorithm's accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) using testing data. We recruited 245 patients who completed 1319 PRO assessments. The final voting ensemble produced state-of-the-art results on the task of predicting 180-day mortality for ovarian cancer paitents (Accuracy = 0.79, Sensitivity = 0.71, Specificity = 0.80, AUROC = 0.76). The algorithm correctly identified 25 of the 35 women in the testing dataset who died within 180 days of assessment. Machine learning algorithms trained using PRO data offer encouraging performance in predicting whether a woman with ovarian cancer will die within 180 days. This model could be used to drive data-driven end-of-life care and address current shortcomings in care delivery. Our model demonstrates the potential of biopsychosocial PROM information to make substantial contributions to oncology prediction modeling. This model could inform clinical decision-making Future research is needed to validate these findings in a larger, more diverse sample.

show abstract

Section: Discussionmentioning

confidence: 99%

Predicting 180-day mortality for women with ovarian cancer using machine learning and patient-reported outcome data

Sidey‐Gibbons

Sun

Schneider

et al. 2022

Sci Rep

View full text Add to dashboard Cite

show abstract

“…The Fair-Net architecture expands the Cal-Net architecture, which developed to improve calibration on imbalanced datasets [14]. Like the Cal-Net architecture, the Fair-Net architecture transforms the binary classification problem into a multi-task problem using two outputs (Figure 1).…”

Section: A the Fair-net Architecturementioning

confidence: 99%

“…We also use the histogram loss from Cal-Net [14] on the primary output Y for generating well scaled-probabilities. In a well-calibrated probabilistic model for binary classification tasks, the proportion of positive examples in each bin of a reliability diagram should match the average of the predictions for the bin, which is usually close to the midpoint of the bin.…”

Section: B Loss Componentsmentioning

confidence: 99%

“…3) With and without histogram loss (L H ): We introduced two variants of Fair-Nets that used the histogram loss [14]. Our case studies across COMPAS data, credit card default dataset and adult census dataset showed that the variants of Fair-Nets trained using histogram loss often outperformed the corresponding variants of Fair-Nets without the histogram loss by achieving lower ECE scores.…”

Section: Ablation Analysesmentioning

confidence: 99%

“…Little work has been done to develop neural network architecture that jointly learns classification and calibration in under-represented sub-populations while handling the skewed distribution of the minority and majority samples in class imbalanced datasets. Recently, the Cal-Net neural network architecture [14] demonstrated simultaneous improvement in classification and calibration performance on class imbalanced datasets. Here, we aim to build on this architecture to improve predictive performance across multiple sub-populations with Fair-Net: a neural network architecture that simultaneously optimizes classification and calibration performances across identifiable sub-populations in the dataset population.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fair-Net: A Network Architecture For Reducing Performance Disparity Between Identifiable Sub-Populations

Datta¹,

Swamidass²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In real world datasets, particular groups are underrepresented, much rarer than others, and machine learning classifiers will often preform worse on under-represented populations. This problem is aggravated across many domains where datasets are class imbalanced, with a minority class far rarer than the majority class. Naive approaches to handle under-representation and class imbalance include training sub-population specific classifiers that handle class imbalance or training a global classifier that overlooks sub-population disparities and aims to achieve high overall accuracy by handling class imbalance. In this study, we find that these approaches are vulnerable in class imbalanced datasets with minority sub-populations. We introduced Fair-Net, a branched multitask neural network architecture that improves both classification accuracy and probability calibration across identifiable sub-populations in class imbalanced datasets. Fair-Nets is a straightforward extension to the output layer and error function of a network, so can be incorporated in far more complex architectures. Empirical studies with three real world benchmark datasets demonstrate that Fair-Net improves classification and calibration performance, substantially reducing performance disparity between gender and racial subpopulations.

show abstract

Calibration methods in imbalanced binary classification

Guilbert,

Caelen,

Chirita

et al. 2024

Ann Math Artif Intell

View full text Add to dashboard Cite

Cal-Net: Jointly Learning Classification and Calibration On Imbalanced Binary Classification Tasks

Cited by 8 publications

References 20 publications

Predicting 180-day mortality for women with ovarian cancer using machine learning and patient-reported outcome data

Predicting 180-day mortality for women with ovarian cancer using machine learning and patient-reported outcome data

Fair-Net: A Network Architecture For Reducing Performance Disparity Between Identifiable Sub-Populations

Calibration methods in imbalanced binary classification

Contact Info

Product

Resources

About