Fairness Violations and Mitigation under Covariate Shift

Singh, Harvineet; Singh, Rina; Mhasawade, Vishwali; Chunara, Rumi

doi:10.1145/3442188.3445865

Cited by 57 publications

(60 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work has also proposed methodological advances to ensure transferability of models across settings, for example, by pre-training on large datasets from related machine learning tasks 62,63 and with the help of causal knowledge about shifts. [64][65][66] Better metrics for dataset shift can help practitioners decide whether to transfer a model to a new setting based on how large the shift is between hospitals, for example. Importantly, findings here showed that the race variable often mediated shifts in clinical variables.…”

Section: Discussionmentioning

confidence: 99%

“…We hope that current work motivates development and evaluation of such metrics on larger and more diverse populations and datasets. Recent work has also proposed methodological advances to ensure transferability of models across settings, for example, by pre-training on large datasets from related machine learning tasks (63,64) and with the help of causal knowledge about shifts (65)(66)(67). Better metrics for dataset shift can help practitioners decide whether to transfer a model to a new setting based on how large the shift is between hospitals, for example.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

Singh

Mhasawade

Chunara

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Importance: Modern predictive models require large amounts of data for training and evaluation which can result in building models that are specific to certain locations, populations in them and clinical practices. Yet, best practices and guidelines for clinical risk prediction models have not yet considered such challenges to generalizability. Objectives: To investigate changes in measures of predictive discrimination, calibration, and algorithmic fairness when transferring models for predicting in-hospital mortality across ICUs in different populations. Also, to study the reasons for the lack of generalizability in these measures. Design, Setting, and Participants: In this multi-center cross-sectional study, electronic health records from 179 hospitals across the US with 70,126 hospitalizations were analyzed. Time of data collection ranged from 2014 to 2015. Main Outcomes and Measures: The main outcome is in-hospital mortality. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for discrimination and calibration metrics, namely area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm "Fast Causal Inference" (FCI) that infers paths of causal influence while identifying potential influences associated with unmeasured variables. Results: In-hospital mortality rates differed in the range of 3.9%-9.3% (1st-3rd quartile) across hospitals. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st to 3rd quartile; median 0.801); calibration slope from 0.725 to 0.983 (1st to 3rd quartile; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (1st to 3rd quartile; median 0.092). When transferring models across geographies, AUC ranged from 0.795 to 0.813 (1st to 3rd quartile; median 0.804); calibration slope from 0.904 to 1.018 (1st to 3rd quartile; median 0.968); and disparity in false negative rates from 0.018 to 0.074 (1st to 3rd quartile; median 0.040). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. Shifts in the race variable distribution and some clinical (vitals, labs and surgery) variables by hospital or region. Race variable also mediates differences in the relationship between clinical variables and mortality, by hospital/region. Conclusions and Relevance: Group-specific metrics should be assessed during generalizability checks to identify potential harms to the groups. In order to develop methods to improve and guarantee performance of prediction models in new environments for groups and individuals, better understanding and provenance of health processes as well as data generating processes by sub-group are needed to identify and mitigate sources of variation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

Singh

Mhasawade

Chunara

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…As mentioned in previous section, the context of fairness is domain dependent and there maybe inherent trade-off in aggregating multiple metrics in different use cases [Kleinberg et al, 2016, Singh et al, 2021.…”

Section: Desired Characteristics Of Fairness Metricsmentioning

confidence: 99%

Fairness Score and Process Standardization: Framework for Fairness Certification in Artificial Intelligence Systems

Agarwal,

Agarwal

2022

Preprint

View full text Add to dashboard Cite

Decisions made by various Artificial Intelligence (AI) systems greatly influence our day-to-day lives. With the increasing use of AI systems, it becomes crucial to know that they are fair, identify the underlying biases in their decision-making, and create a standardized framework to ascertain their fairness. In this paper, we propose a novel Fairness Score to measure the fairness of a data-driven AI system and a Standard Operating Procedure (SOP) for issuing Fairness Certification for such systems. Fairness Score and audit process standardization will ensure quality, reduce ambiguity, enable comparison and improve the trustworthiness of the AI systems. It will also provide a framework to operationalise the concept of fairness and facilitate the commercial deployment of such systems. Furthermore, a Fairness Certificate issued by a designated third-party auditing agency following the standardized process would boost the conviction of the organizations in the AI systems that they intend to deploy. The Bias Index proposed in this paper also reveals comparative bias amongst the various protected attributes within the dataset. To substantiate the proposed framework, we iteratively train a model on biased and unbiased data using multiple datasets and check that the Fairness Score and the proposed process correctly identify the biases and judge the fairness.

show abstract

“…In on-device ML settings, trained ML models undergo multiple post-processing steps to overcome resource constraints for on-device deployment and distribution shifts due to context heterogeneity. Some of these post-processing steps, like domain adaptation [39] and model compression [21], can be biased. Rather than looking at the compound effect of multiple algorithmic decisions, we consider the propagation of bias through the different processing stages in the on-device ML development pipeline.…”

Section: Related Workmentioning

confidence: 99%

Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows

Toussaint¹,

Mathur²,

Kawsar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Billions of distributed, heterogeneous and resource constrained smart consumer devices deploy on-device machine learning (ML) to deliver private, fast and offline inference on personal data. On-device ML systems are highly context dependent, and sensitive to user, usage, hardware and environmental attributes. Despite this sensitivity and the propensity towards bias in ML, bias in on-device ML has not been studied. This paper studies the propagation of bias through design choices in on-device ML development workflows. We position reliablity bias, which arises from disparate device failures across demographic groups, as a source of unfairness in on-device ML settings and quantify metrics to evaluate it. We then identify complex and interacting technical design choices in the on-device ML workflow that can lead to disparate performance across user groups, and thus reliability bias. Finally, we show with an empirical case study that seemingly innocuous design choices such as the data sample rate, pre-processing parameters used to construct input features and pruning hyperparameters propagate reliability bias through an audio keyword spotting development workflow. We leverage our insights to suggest strategies for developers to develop fairer on-device ML. CCS Concepts: • Hardware → Analysis and design of emerging devices and systems; • Computing methodologies → Machine learning; • General and reference → Reliability.

show abstract

Fairness Violations and Mitigation under Covariate Shift

Cited by 57 publications

References 27 publications

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

Fairness Score and Process Standardization: Framework for Fairness Certification in Artificial Intelligence Systems

Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows

Contact Info

Product

Resources

About