Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare

Pfohl, Stephen R.; Xu, Yizhe; Foryciarz, Agata; Ignatiadis, Nikolaos; Genkins, Julian Z.; Shah, Nigam H.

doi:10.1145/3531146.3533166

Cited by 18 publications

(17 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The discrimination power of all four models are lower in patients with comorbidity whose event rates are nearly twice as high as those without comorbidity. Such differential performance across subgroups is referred to as metric disparity in model fairness assessment, and is necessary to address to mitigate health inequalities [41]. As expected, such metric disparity is exacerbated when the number of subgroups grows larger, such as across 299 geographic locations.…”

Section: Discussionmentioning

confidence: 99%

Clinical Utility Gains from Incorporating Comorbidity and Geographic Location Information into Risk Estimation Equations for Atherosclerotic Cardiovascular Disease

Xu¹,

Foryciarz²,

Steinberg³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Objective: There are several efforts to re-learn the 2013 ACC/AHA pooled cohort equations (PCE) for patients with specific comorbidities and geographic locations. With over 363 customized risk models in the literature, we aim to evaluate such revised models to determine if the performance improvements translate to gains in clinical utility.Methods: We re-train a baseline PCE using the ACC/AHA PCE variables and revise it to incorporate subject-level geographic location and comorbidity information. We apply fixed effects, random effects, and extreme gradient boosting models to handle the correlation and heterogeneity induced by locations. Models are trained using 2,464,522 claims records from Optum © 's Clinformatics ® Data Mart and validated in the hold-out set (N=1,056,224). We evaluate models' performance overall and across subgroups defined by the presence or absence of chronic kidney disease (CKD) or rheumatoid arthritis (RA) and geographic locations. We evaluate models' expected net benefit using decision curve analysis and models' statistical properties using several discrimination and calibration metrics.Results: The baseline PCE is miscalibrated overall, in patients with CKD or RA, and locations with small populations. Our revised models improved both the overall (GND P-value=0.41) and subgroup calibration but only enhanced net benefit in the underrepresented subgroups. The gains are larger in the subgroups with comorbidities and heterogeneous across geographic locations.Conclusions: Revising the PCE with comorbidity and location information significantly enhanced models' calibration; however, such improvements do not necessarily translate to clinical gains. Thus, we recommend future works to quantify the consequences from using risk calculators to guide clinical decisions.

show abstract

Section: Discussionmentioning

confidence: 99%

Clinical Utility Gains from Incorporating Comorbidity and Geographic Location Information into Risk Estimation Equations for Atherosclerotic Cardiovascular Disease

Xu¹,

Foryciarz²,

Steinberg³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To begin, what is right or wrong, or what is desired behavior, is not always agreed upon by all stakeholders. This has been discussed at length in the fairness literature [19,58,59,73,83,97,108,123,148,155,181,243,250] and in practice leads to uncertainty regarding measurement and evaluation. Research on operationalizing and comparing different definitions and measures of fair or ethical behavior makes up a significant percentage of papers published at FAccT and similar conferences.…”

Section: System Evaluation and Measurementmentioning

confidence: 99%

Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities

Nashed¹,

Svegliato²,

Blodgett³

2023

Preprint

View full text Add to dashboard Cite

As automated decision making and decision assistance systems become common in everyday life, research on the prevention or mitigation of potential harms that arise from decisions made by these systems has proliferated. However, various research communities have independently conceptualized these harms, envisioned potential applications, and proposed interventions. The result is a somewhat fractured landscape of literature focused generally on ensuring decision-making algorithms "do the right thing". In this paper, we compare and discuss work across two major subsets of this literature: algorithmic fairness, which focuses primarily on predictive systems, and ethical decision making, which focuses primarily on sequential decision making and planning. We explore how each of these settings has articulated its normative concerns, the viability of different techniques for these different settings, and how ideas from each setting may have utility for the other.Finally, given the optimal value function * ( ), the optimal policy * ( ) can be calculated in the following way: * ( ) = arg max ∈ * ( ).

show abstract

“…While the fairness of risk models has been widely discussed in the literature, this has been predominantly done in the context of risk scores used for automated decision-making, i.e., discrete classi cations derived from risk scores based on a xed threshold [11,23,34,41,45,75,89]. As a consequence, many of the proposed standard fairness metrics such as equalized odds, equal opportunity, parity of predictive values, or net bene t, are a function of classi cation decisions, and not of the risk score itself [34,75]. This can be problematic for a number of reasons.…”

Section: Background and Related Workmentioning

confidence: 99%

“…We thus argue here that the fairness of a risk model is fundamentally determined by the epistemic value it provides to the di erent groups, and we consider a risk score model fair if it provides similar epistemic value to all groups of interest. This di ers fundamentally from the classi cation case, in which fairness is typically characterized in terms of the relative costs or bene ts provided to di erent groups as a result of decisions made based on the model, such as in equalized odds [34], equality of opportunity [34], or the expected net bene t of an intervention for di erent groups [75].…”

Section: Fairness Desiderata For Risk Score Modelsmentioning

confidence: 99%

On (assessing) the fairness of risk score models

Petersen¹,

Ganz²,

Holm³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent work on algorithmic fairness has largely focused on the fairness of discrete decisions, or classi cations. While such decisions are often based on risk score models, the fairness of the risk models themselves has received considerably less attention. Risk models are of interest for a number of reasons, including the fact that they communicate uncertainty about the potential outcomes to users, thus representing a way to enable meaningful human oversight. Here, we address fairness desiderata for risk score models. We identify the provision of similar epistemic value to di erent groups as a key desideratum for risk score fairness. Further, we address how to assess the fairness of risk score models quantitatively, including a discussion of metric choices and meaningful statistical comparisons between groups. In this context, we also introduce a novel calibration error metric that is less sample size-biased than previously proposed metrics, enabling meaningful comparisons between groups of di erent sizes. We illustrate our methodology -which is widely applicable in many other settings -in two case studies, one in recidivism risk prediction, and one in risk of major depressive disorder (MDD) prediction.

show abstract

Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare

Cited by 18 publications

References 45 publications

Clinical Utility Gains from Incorporating Comorbidity and Geographic Location Information into Risk Estimation Equations for Atherosclerotic Cardiovascular Disease

Clinical Utility Gains from Incorporating Comorbidity and Geographic Location Information into Risk Estimation Equations for Atherosclerotic Cardiovascular Disease

Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities

On (assessing) the fairness of risk score models

Contact Info

Product

Resources

About