Tackling bias in AI health datasets through the STANDING Together initiative

Ganapathi, Shaswath; Palmer, Jo; Alderman, Joseph; Calvert, Melanie; Espinoza, Cyrus; Gath, Jacqui; Ghassemi, Marzyeh; Heller, Katherine; McKay, Francis; Karthikesalingam, Alan; Kuku, Stephanie; Mackintosh, Maxine; Manohar, Sinduja; Mateen, Bilal A.; Matin, Rubeta; McCradden, Melissa D; Oakden‐Rayner, Lauren; Ordish, Johan; Pearson, Russell J.; Pfohl, Stephen R.; Rostamzadeh, Negar; Sapey, Elizabeth; Sebire, Neil J; Sounderajah, Viknesh; Summers, Charlotte; Treanor, Darren; Denniston, Alastair K; Liu, Xiaoxuan

doi:10.1038/s41591-022-01987-w

Cited by 44 publications

(33 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results are surprising given racial categories are imprecise, lack objective definitions, and have varied over time and by geography 25 . Nonetheless, identifying if and when AI may predict race is important for the thoughtful development of equitable applications of AI to medicine 26 . We believe it is equally important to understand how and why AI may predict race.…”

Section: Discussionmentioning

confidence: 99%

Confounders mediate AI prediction of demographics in medical imaging

et al. 2022

View full text Add to dashboard Cite

Deep learning has been shown to accurately assess “hidden” phenotypes from medical imaging beyond traditional clinician interpretation. Using large echocardiography datasets from two healthcare systems, we test whether it is possible to predict age, race, and sex from cardiac ultrasound images using deep learning algorithms and assess the impact of varying confounding variables. Using a total of 433,469 videos from Cedars-Sinai Medical Center and 99,909 videos from Stanford Medical Center, we trained video-based convolutional neural networks to predict age, sex, and race. We found that deep learning models were able to identify age and sex, while unable to reliably predict race. Without considering confounding differences between categories, the AI model predicted sex with an AUC of 0.85 (95% CI 0.84–0.86), age with a mean absolute error of 9.12 years (95% CI 9.00–9.25), and race with AUCs ranging from 0.63 to 0.71. When predicting race, we show that tuning the proportion of confounding variables (age or sex) in the training data significantly impacts model AUC (ranging from 0.53 to 0.85), while sex and age prediction was not particularly impacted by adjusting race proportion in the training dataset AUC of 0.81–0.83 and 0.80–0.84, respectively. This suggests significant proportion of AI’s performance on predicting race could come from confounding features being detected. Further work remains to identify the particular imaging features that associate with demographic information and to better understand the risks of demographic identification in medical AI as it pertains to potentially perpetuating bias and disparities.

show abstract

Section: Discussionmentioning

confidence: 99%

Confounders mediate AI prediction of demographics in medical imaging

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Considering that data from these populations may be used to develop algorithms for patients from outside these regions, it is almost expected that postvalidation real-world model performance would be substandard and that such models incorporate the biases inherent to their training and validation datasets. Initiatives such as the STANDING Together artificial intelligence initiative, which aim to standardize dataset curation, identify and map dataset deficiencies in priority disease areas, and envelop dataset curators into the development process, are an excellent step forward to addressing digital health data sparsity [26]. Within ophthalmology specifically, the American Academy of Ophthalmology's Intelligent Research in Sight Registry also offers clinicians and researchers an accessible repository for ocular data ready for research use, albeit with the caveat of geographic limitation to the United States [27].…”

Section: Approaches To Improve Model Validationmentioning

confidence: 99%

Toward safer ophthalmic artificial intelligence via distributed validation on real-world data

Nath

Rahimy

Kras

et al. 2023

Current Opinion in Ophthalmology

View full text Add to dashboard Cite

Purpose of review The current article provides an overview of the present approaches to algorithm validation, which are variable and largely self-determined, as well as solutions to address inadequacies. Recent findings In the last decade alone, numerous machine learning applications have been proposed for ophthalmic diagnosis or disease monitoring. Remarkably, of these, less than 15 have received regulatory approval for implementation into clinical practice. Although there exists a vast pool of structured and relatively clean datasets from which to develop and test algorithms in the computational ‘laboratory’, real-world validation remains key to allow for safe, equitable, and clinically reliable implementation. Bottlenecks in the validation process stem from a striking paucity of regulatory guidance surrounding safety and performance thresholds, lack of oversight on critical postdeployment monitoring and context-specific recalibration, and inherent complexities of heterogeneous disease states and clinical environments. Implementation of secure, third-party, unbiased, pre and postdeployment validation offers the potential to address existing shortfalls in the validation process. Summary Given the criticality of validation to the algorithm pipeline, there is an urgent need for developers, machine learning researchers, and end-user clinicians to devise a consensus approach, allowing for the rapid introduction of safe, equitable, and clinically valid machine learning implementations.

show abstract

“…[1][2][3][4] While there is consensus that data flow between organizations and developers is necessary to develop AI algorithms, many organizations remain reluctant to share health data for AI due to organizational apprehensions and competing priorities, affecting equitable AI development in the health care sector. [5][6][7][8][9] Current research highlights problematic trends in health data sharing. A systematic review by Kaushal et al 10 highlighted that large data-sharing initiatives are often performed by large academic institutions from a few geographic locations with access to funding and the technical expertise to transform and share well-curated health data sets responsibly.…”

Section: Introductionmentioning

confidence: 99%

“…Advances in algorithm development have shown that artificial intelligence (AI) and machine learning can augment clinical decision-making, promoting diagnostic excellence . While there is consensus that data flow between organizations and developers is necessary to develop AI algorithms, many organizations remain reluctant to share health data for AI due to organizational apprehensions and competing priorities, affecting equitable AI development in the health care sector …”

Section: Introductionmentioning

confidence: 99%

Organizational Factors in Clinical Data Sharing for Artificial Intelligence in Health Care

Youssef,

Ng,

Long

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceLimited sharing of data sets that accurately represent disease and patient diversity limits the generalizability of artificial intelligence (AI) algorithms in health care.ObjectiveTo explore the factors associated with organizational motivation to share health data for AI development.Design, Setting, and ParticipantsThis qualitative study investigated organizational readiness for sharing health data across the academic, governmental, nonprofit, and private sectors. Using a multiple case studies approach, 27 semistructured interviews were conducted with leaders in data-sharing roles from August 29, 2022, to January 9, 2023. The interviews were conducted in the English language using a video conferencing platform. Using a purposive and nonprobabilistic sampling strategy, 78 individuals across 52 unique organizations were identified. Of these, 35 participants were enrolled. Participant recruitment concluded after 27 interviews, as theoretical saturation was reached and no additional themes emerged.Main Outcome and MeasureConcepts defining organizational readiness for data sharing and the association between data-sharing factors and organizational behavior were mapped through iterative qualitative analysis to establish a framework defining organizational readiness for sharing clinical data for AI development.ResultsInterviews included 27 leaders from 18 organizations (academia: 10, government: 7, nonprofit: 8, and private: 2). Organizational readiness for data sharing centered around 2 main constructs: motivation and capabilities. Motivation related to the alignment of an organization’s values with data-sharing priorities and was associated with its engagement in data-sharing efforts. However, organizational motivation could be modulated by extrinsic incentives for financial or reputational gains. Organizational capabilities comprised infrastructure, people, expertise, and access to data. Cross-sector collaboration was a key strategy to mitigate barriers to access health data.Conclusions and RelevanceThis qualitative study identified sector-specific factors that may affect the data-sharing behaviors of health organizations. External incentives may bolster cross-sector collaborations by helping overcome barriers to accessing health data for AI development. The findings suggest that tailored incentives may boost organizational motivation and facilitate sustainable flow of health data for AI development.

show abstract

Tackling bias in AI health datasets through the STANDING Together initiative

Cited by 44 publications

References 9 publications

Confounders mediate AI prediction of demographics in medical imaging

Confounders mediate AI prediction of demographics in medical imaging

Toward safer ophthalmic artificial intelligence via distributed validation on real-world data

Organizational Factors in Clinical Data Sharing for Artificial Intelligence in Health Care

Contact Info

Product

Resources

About