The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations

Balagopalan, Aparna; Zhang, Haoran; Hamidieh, Kimia; Hartvigsen, Thomas; Rudzicz, Frank; Ghassemi, Marzyeh

doi:10.1145/3531146.3533179

Cited by 31 publications

(9 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the U.S. Food and Drug Administration (FDA) recently recommended that the deployment of any AI-based medical device used to inform human decisions must address "human factors considerations and the human interpretability of model inputs" 35 . While increasing model interpretability is an appealing approach to humans, existing approaches to interpretability and explainability are poorly suited to health care 36 , may decrease human ability to identify model mistakes 7 , and increase model bias (i.e., the gap in model performance between the worst and best subgroup) 37 . Any successful deployment must thus rigorously test and validate several human-AI recommendation styles to ensure that AI systems are substantially improving decision making.…”

Section: Discussionmentioning

confidence: 99%

Mitigating the impact of biased artificial intelligence in emergency decision-making

et al. 2022

Self Cite

View full text Add to dashboard Cite

Background Prior research has shown that artificial intelligence (AI) systems often encode biases against minority subgroups. However, little work has focused on ways to mitigate the harm discriminatory algorithms can cause in high-stakes settings such as medicine. Methods In this study, we experimentally evaluated the impact biased AI recommendations have on emergency decisions, where participants respond to mental health crises by calling for either medical or police assistance. We recruited 438 clinicians and 516 non-experts to participate in our web-based experiment. We evaluated participant decision-making with and without advice from biased and unbiased AI systems. We also varied the style of the AI advice, framing it either as prescriptive recommendations or descriptive flags. Results Participant decisions are unbiased without AI advice. However, both clinicians and non-experts are influenced by prescriptive recommendations from a biased algorithm, choosing police help more often in emergencies involving African-American or Muslim men. Crucially, using descriptive flags rather than prescriptive recommendations allows respondents to retain their original, unbiased decision-making. Conclusions Our work demonstrates the practical danger of using biased models in health contexts, and suggests that appropriately framing decision support can mitigate the effects of AI bias. These findings must be carefully considered in the many real-world clinical scenarios where inaccurate or biased models may be used to inform important decisions.

show abstract

Section: Discussionmentioning

confidence: 99%

Mitigating the impact of biased artificial intelligence in emergency decision-making

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Second, it has also remained unknown whether AIbased explanations exhibit a bias, where their reliability differs across surgeon sub-cohorts. Although preliminary studies have begun to explore the intersection of bias and explanations 26,27,43 , they do not leverage human expert explanations, are limited to non-surgical domains, and do not present findings for video-based AI systems. Third, the development of a strategy that consistently improves the reliability and fairness of explanations has been underexplored.…”

Section: Discussionmentioning

confidence: 99%

“…However, these studies remain qualitative and thereby do not systematically investigate whether explanations are consistently reliable across data points. Studies that quantitatively evaluate AI-based explanations often exclude a comparison to human explanations 24,25 , a drawback that extends to the preliminary studies aimed at also assessing the fairness of such explanations 26,27 . Notably, previous work has not quantitatively compared AI-based explanations to human explanations in the context of surgical videos, nor has it proposed a strategy to enhance the reliability and fairness of such explanations.…”

mentioning

confidence: 99%

A multi-institutional study using artificial intelligence to provide reliable and fair feedback to surgeons

et al. 2023

View full text Add to dashboard Cite

Background Surgeons who receive reliable feedback on their performance quickly master the skills necessary for surgery. Such performance-based feedback can be provided by a recently-developed artificial intelligence (AI) system that assesses a surgeon’s skills based on a surgical video while simultaneously highlighting aspects of the video most pertinent to the assessment. However, it remains an open question whether these highlights, or explanations, are equally reliable for all surgeons. Methods Here, we systematically quantify the reliability of AI-based explanations on surgical videos from three hospitals across two continents by comparing them to explanations generated by humans experts. To improve the reliability of AI-based explanations, we propose the strategy of training with explanations –TWIX –which uses human explanations as supervision to explicitly teach an AI system to highlight important video frames. Results We show that while AI-based explanations often align with human explanations, they are not equally reliable for different sub-cohorts of surgeons (e.g., novices vs. experts), a phenomenon we refer to as an explanation bias. We also show that TWIX enhances the reliability of AI-based explanations, mitigates the explanation bias, and improves the performance of AI systems across hospitals. These findings extend to a training environment where medical students can be provided with feedback today. Conclusions Our study informs the impending implementation of AI-augmented surgical training and surgeon credentialing programs, and contributes to the safe and fair democratization of surgery.

show abstract

“…The concept of technical robustness is an important cornerstone for ensuring Trustworthy AI. The improvement of balanced and robust training techniques and datasets can enhance not only fairness (see Section 4.4) but also explainability (see Section 4.2) [110].…”

Section: Technical Robustness and Generalizationmentioning

confidence: 99%

A Survey on the Use of Synthetic Data for Enhancing Key Aspects of Trustworthy AI in the Energy Domain: Challenges and Opportunities

Meiser,

Zinnikus

2024

Energies

View full text Add to dashboard Cite

To achieve the energy transition, energy and energy efficiency are becoming more and more important in society. New methods, such as Artificial Intelligence (AI) and Machine Learning (ML) models, are needed to coordinate supply and demand and address the challenges of the energy transition. AI and ML are already being applied to a growing number of energy infrastructure applications, ranging from energy generation to energy forecasting and human activity recognition services. Given the rapid development of AI and ML, the importance of Trustworthy AI is growing as it takes on increasingly responsible tasks. Particularly in the energy domain, Trustworthy AI plays a decisive role in designing and implementing efficient and reliable solutions. Trustworthy AI can be considered from two perspectives, the Model-Centric AI (MCAI) and the Data-Centric AI (DCAI) approach. We focus on the DCAI approach, which relies on large amounts of data of sufficient quality. These data are becoming more and more synthetically generated. To address this trend, we introduce the concept of Synthetic Data-Centric AI (SDCAI). In this survey, we examine Trustworthy AI within a Synthetic Data-Centric AI context, focusing specifically on the role of simulation and synthetic data in enhancing the level of Trustworthy AI in the energy domain.

show abstract

The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations

Cited by 31 publications

References 54 publications

Mitigating the impact of biased artificial intelligence in emergency decision-making

Mitigating the impact of biased artificial intelligence in emergency decision-making

A multi-institutional study using artificial intelligence to provide reliable and fair feedback to surgeons

A Survey on the Use of Synthetic Data for Enhancing Key Aspects of Trustworthy AI in the Energy Domain: Challenges and Opportunities

Contact Info

Product

Resources

About