Critical Bias in Critical Care Devices

Charpignon, Marie-Laure; Byers, Joseph; Cabral, Stephanie; Celi, Leo Anthony; Fernandes, Chrystinne Oliveira; Gallifant, Jack; Lough, Mary E.; Mlombwa, Donald; Moukheiber, Lama; Ong, Bradley Ashley G; Panitchote, Anupol; William, Wasswa; Wong, An-Kwok Ian; Nazer, Lama

doi:10.1016/j.ccc.2023.02.005

Cited by 20 publications

(13 citation statements)

References 90 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The algorithmic bias in OPE is a trade-off with variation but is inherent to this evaluation method (3). Furthermore, in state-spaces derived from electronic health record data, biases arising from medical devices and social determinants of care can result in inaccurate agent recommendations, stemming from misrepresented states, comparable to supervised machine learning (59). These data could in the future be incorporated in the state-space if also the clinician's notes are incorporated in the dataset (60).…”

Section: Discussionmentioning

confidence: 99%

Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

Otten,

Jagesar,

Dam

et al. 2023

Critical Care Medicine

View full text Add to dashboard Cite

Objective: Reinforcement learning (RL) is a machine learning technique uniquely effective at sequential decision-making, which makes it potentially relevant to ICU treatment challenges. We set out to systematically review, assess level-of-readiness and meta-analyze the effect of RL on outcomes for critically ill patients. Data Sources: A systematic search was performed in PubMed, Embase.com, Clarivate Analytics/Web of Science Core Collection, Elsevier/SCOPUS and the Institute of Electrical and Electronics Engineers Xplore Digital Library from inception to March 25, 2022, with subsequent citation tracking. Data Extraction: Journal articles that used an RL technique in an ICU population and reported on patient health-related outcomes were included for full analysis. Conference papers were included for level-of-readiness assessment only. Descriptive statistics, characteristics of the models, outcome compared with clinician’s policy and level-of-readiness were collected. RL-health risk of bias and applicability assessment was performed. Data Synthesis: A total of 1,033 articles were screened, of which 18 journal articles and 18 conference papers, were included. Thirty of those were prototyping or modeling articles and six were validation articles. All articles reported RL algorithms to outperform clinical decision-making by ICU professionals, but only in retrospective data. The modeling techniques for the state-space, action-space, reward function, RL model training, and evaluation varied widely. The risk of bias was high in all articles, mainly due to the evaluation procedure. Conclusion: In this first systematic review on the application of RL in intensive care medicine we found no studies that demonstrated improved patient outcomes from RL-based technologies. All studies reported that RL-agent policies outperformed clinician policies, but such assessments were all based on retrospective off-policy evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

Otten,

Jagesar,

Dam

et al. 2023

Critical Care Medicine

View full text Add to dashboard Cite

show abstract

“…Specifically, pulse oximeters exhibit reduced accuracy in patients with darker skin pigmentation, an issue attributed to device miscalibration and the lack of diversity in development phases [106,116,127,133]. Similar disparities in device accuracy affecting measurements like oxygen saturation, body temperature, and blood pressure [44] have been documented, often resulting from insufficiently diverse calibration populations [26]. A systematic review of mechanical ventilation studies found that AI applied to mechanical ventilation has limited external validation and model calibration, with a substantial risk of bias, significant gaps in reporting, and poor code and data availability [53] Such discrepancies in device performance can introduce biases into clinical data, potentially influencing treatment decisions, such as the administration of supplemental oxygen or the preference for certain temperature measurement methods, thereby affecting diagnoses and treatments for specific racial subgroups.…”

Section: Data Bias In Medical Devices and Algorithmsmentioning

confidence: 97%

Inherent Bias in Electronic Health Records: A Scoping Review of Sources of Bias

Perets,

Stagno,

Yehuda

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Objectives: Biases inherent in electronic health records (EHRs), and therefore in medical artificial intelligence (AI) models may significantly exacerbate health inequities and challenge the adoption of ethical and responsible AI in healthcare. Biases arise from multiple sources, some of which are not as documented in the literature. Biases are encoded in how the data has been collected and labeled, by implicit and unconscious biases of clinicians, or by the tools used for data processing. These biases and their encoding in healthcare records undermine the reliability of such data and bias clinical judgments and medical outcomes. Moreover, when healthcare records are used to build data-driven solutions, the biases are further exacerbated, resulting in systems that perpetuate biases and induce healthcare disparities. This literature scoping review aims to categorize the main sources of biases inherent in EHRs. Methods: We queried PubMed and Web of Science on January 19th, 2023, for peer-reviewed sources in English, published between 2016 and 2023, using the PRISMA approach to stepwise scoping of the literature. To select the papers that empirically analyze bias in EHR, from the initial yield of 430 papers, 27 duplicates were removed, and 403 studies were screened for eligibility. 196 articles were removed after the title and abstract screening, and 96 articles were excluded after the full-text review resulting in a final selection of 116 articles. Results: Systematic categorizations of diverse sources of bias are scarce in the literature, while the effects of separate studies are often convoluted and methodologically contestable. Our categorization of published empirical evidence identified the six main sources of bias: a) bias arising from past clinical trials; b) data-related biases arising from missing, incomplete information or poor labeling of data; human-related bias induced by c) implicit clinician bias, d) referral and admission bias; e) diagnosis or risk disparities bias and finally, (f) biases in machinery and algorithms. Conclusions: Machine learning and data-driven solutions can potentially transform healthcare delivery, but not without limitations. The core inputs in the systems (data and human factors) currently contain several sources of bias that are poorly documented and analyzed for remedies. The current evidence heavily focuses on data-related biases, while other sources are less often analyzed or anecdotal. However, these different sources of biases add to one another exponentially. Therefore, to understand the issues holistically we need to explore these diverse sources of bias. While racial biases in EHR have been often documented, other sources of biases have been less frequently investigated and documented (e.g. gender-related biases, sexual orientation discrimination, socially induced biases, and implicit, often unconscious, human-related cognitive biases). Moreover, some existing studies lack causal evidence, illustrating the different prevalences of disease across groups, which does not per se prove the causality. Our review shows that data-, human- and machine biases are prevalent in healthcare and they significantly impact healthcare outcomes and judgments and exacerbate disparities and differential treatment. Understanding how diverse biases affect AI systems and recommendations is critical. We suggest that researchers and medical personnel should develop safeguards and adopt data-driven solutions with a "bias-in-mind" approach. More empirical evidence is needed to tease out the effects of different sources of bias on health outcomes.

show abstract

“…Pulse oximetry is a prominent example of how racial and ethnic bias can manifest in critical care medical equipment [46]. Underperformance of the pulse oximeter in patients with darker skin color has been shown to result in events of hidden hypoxemia, which can be defined as SaO 2 (measured by arterial blood gas [ABG]) < 88%, but SpO 2 (measured by pulse oximetry) ≥ 92% [45, 46].…”

Section: Example Case Studymentioning

confidence: 99%

Participant Flow Diagrams for Health Equity in AI Research

Ellen,

Matos,

Viola

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Biases in sample creation can arise at any study phase, including initial patient recruitment, exclusion criteria, input-level exclusion and outcome-level exclusion, and often reflect the underrepresentation or exclusion of demographic groups historically disadvantaged in medical research. The use of non-representative samples to construct clinical algorithms in artificial intelligence (AI) and machine learning (ML) applications may further amplify this selection bias. Building on the “Data Cards” initiative for transparency in AI research, we advocate for the addition of a detailed participant flow diagram for AI studies, emphasizing the need to detail excluded participant demographic characteristics at every study phase. This tracking of excluded participants enhances understanding of potential algorithmic biases before their clinical implementation, and thus deserves to be detailed in any medical AI study. We include both a model for this flow diagram as well as a brief case study explaining how it could be implemented in practice. Through standardized reporting of participant flow diagrams, we can better gauge the potential inequity embedded in AI applications, facilitating more reliable and equitable clinical algorithms.

show abstract

Critical Bias in Critical Care Devices

Cited by 20 publications

References 90 publications

Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

Inherent Bias in Electronic Health Records: A Scoping Review of Sources of Bias

Participant Flow Diagrams for Health Equity in AI Research

Contact Info

Product

Resources

About