External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19

Barish, Matthew A.; Bolourani, Siavash; Lau, Lawrence; Shah, Sareen; Zanos, Theodoros P.

doi:10.1038/s42256-020-00254-2

Cited by 57 publications

(42 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…58 Similarly, a COVID-19 mortality prediction study 59 was found to be irreproducible by three independent research groups from different countries. [60][61][62] Given the global, unprecedented public health challenge caused by COVID-19, we strongly encourage medical researchers to follow the trends toward open-source development in the field of ML (which was proclaimed by various luminaries 14 years ago 63 and successfully implemented in important venues). We encourage researchers to expedite a transformation toward a common practice of validating the proposed methodology and results by publishing both code and, whenever possible, anonymized medical data, especially in academic, non-commercial settings.…”

Section: Choosing the Right Task For Ai Modelsmentioning

confidence: 99%

On the role of artificial intelligence in medical imaging of COVID-19

Born¹,

Beymer²,

Rajan³

et al. 2021

Patterns

View full text Add to dashboard Cite

Although a plethora of research articles on AI methods on COVID-19 medical imaging are published, their clinical value remains unclear. We conducted the largest systematic review of the literature addressing the utility of AI in imaging for COVID-19 patient care. By keyword searches on PubMed and preprint servers throughout 2020, we identified 463 manuscripts and performed a systematic meta-analysis to assess their technical merit and clinical relevance. Our analysis evidences a significant disparity between clinical and AI communities, in the focus on both imaging modalities (AI experts neglected CT and ultrasound, favoring X-ray) and performed tasks (71.9% of AI papers centered on diagnosis). The vast majority of manuscripts were found to be deficient regarding potential use in clinical practice, but 2.7% (n = 12) publications were assigned a high maturity level and are summarized in greater detail. We provide an itemized discussion of the challenges in developing clinically relevant AI solutions with recommendations and remedies.THE BIGGER PICTURE During the COVID-19 pandemic, medical imaging (CT, X-ray, ultrasound) has played a key role in addressing the magnified need for speed, low cost, ubiquity, and precision in patient care. The contemporary digitization of medicine and rise of artificial intelligence (AI) induce a quantum leap in medical imaging: AI has proven equipollent to healthcare professionals across a diverse range of tasks, and hopes are high that AI can save time and cost and increase coverage by advancing rapid patient stratification and empowering clinicians. This review bridges medical imaging and AI in the context of COVID-19 and conducts the largest systematic review of the literature in the field. We identify several gaps and evidence significant disparities between clinicians and AI experts and foresee a need for improved, interdisciplinary collaboration to develop robust AI solutions that can be deployed in clinical practice.The key challenges on that roadmap are discussed alongside recommended solutions. ll

show abstract

Section: Choosing the Right Task For Ai Modelsmentioning

confidence: 99%

On the role of artificial intelligence in medical imaging of COVID-19

Born¹,

Beymer²,

Rajan³

et al. 2021

Patterns

View full text Add to dashboard Cite

show abstract

“…This raises concerns about reproducibility and transparency of their studies, as recently argued against a Nature study on breast cancer screening 11 in a “ matters arising” 58 . Similarly, COVID-19 mortality prediction study 59 was found to be irreproducible by three independent research groups from different countries 60,61,62 . Given the global, unprecedented public health challenge caused by COVID-19, we strongly encourage medical researchers to follow the trends toward open-source development in the field of machine learning (which has been proclaimed by various luminaries fourteen years ago 63 and successfully implemented in important venues).…”

Section: Discussionmentioning

confidence: 94%

“…AI literature focus mainly on diagnosis with less coverage on other tasks with higher clinical relevance, such as severity assessment and monitoring. With few exceptions, the authors of the best publications did not release their clinical datasets, raising concerns about reproducibility and transparency of their studies as was recently exemplified for Nature study on breast cancer screening 59,60 and a COVID-19 mortality prediction study 61 that was found irreproducible by three independent research groups from different countries 62,63,64 . Given the global, unprecedented public health challenge caused by COVID-19, we strongly encourage medical researchers to follow the trends toward open-source development in other fields and expedite a transformation toward a common practice of publishing data and code.…”

Section: Discussionmentioning

confidence: 99%

On the Role of Artificial Intelligence in Medical Imaging of COVID-19

Born

Beymer

Rajan

et al. 2020

Preprint

View full text Add to dashboard Cite

During the COVID-19 pandemic, lung imaging takes a key role in addressing the magnified need of speed, cost, ubiquity and precision in medical care. The rise of artificial intelligence induced a quantum leap in medical imaging: AI has now proven equipollent to healthcare professionals in several diseases and the potential to save time, cost and increase coverage. But AI-accelerated medical imaging must still fully demonstrate its ability in remediating diseases such as COVID-19. We identify key use cases of lung imaging for COVID-19, comparing CT, X-Ray and ultrasound imaging from clinical and AI perspectives. We perform a systematic, manual survey of 197 related publications that reveals a disparity in the focus of the AI and clinical communities, caused by data availability and the lack of collaboration, and in modality trends, driven by ubiquity. Last, challenges in AI-acceleration and ways to remediate them are discussed and future research goals are identified.

show abstract

“…In this relatively simple severity and outcome prediction task, and with a small validation sample size and no external model evaluation, the authors have used the XGBoost classifier method to identify three biomarkers [namely, lactic dehydrogenase (LDH), lymphocyte count and high-sensitivity Creactive protein (hs-CRP)] that will allow the prediction of the mortality of COVID-19 patients more than 10 days in advance with reportedly more than 90% accuracy. External evaluation of this result by several other researchers, such as in Barish et al (2020), Giacobbe (2020), Quanjel et al (2020), andDupuis et al (2021), has shown that the results of Yan et al (2020b) have limited clinical utility as it was impossible to replicate the findings and arrive at the same conclusion. If a huge external evaluation problem exists even for simpler problems (such as prediction and forecasting problems), one can only imagine the scale of the problem when using AI-based model for more complicated problems such as those involving images (computer vision-related problems).…”

Section: Discussionmentioning

confidence: 97%

The Promise of AI in Detection, Diagnosis, and Epidemiology for Combating COVID-19: Beyond the Hype

Abdulkareem

Petersen

2021

Front. Artif. Intell.

View full text Add to dashboard Cite

COVID-19 has created enormous suffering, affecting lives, and causing deaths. The ease with which this type of coronavirus can spread has exposed weaknesses of many healthcare systems around the world. Since its emergence, many governments, research communities, commercial enterprises, and other institutions and stakeholders around the world have been fighting in various ways to curb the spread of the disease. Science and technology have helped in the implementation of policies of many governments that are directed toward mitigating the impacts of the pandemic and in diagnosing and providing care for the disease. Recent technological tools, artificial intelligence (AI) tools in particular, have also been explored to track the spread of the coronavirus, identify patients with high mortality risk and diagnose patients for the disease. In this paper, areas where AI techniques are being used in the detection, diagnosis and epidemiological predictions, forecasting and social control for combating COVID-19 are discussed, highlighting areas of successful applications and underscoring issues that need to be addressed to achieve significant progress in battling COVID-19 and future pandemics. Several AI systems have been developed for diagnosing COVID-19 using medical imaging modalities such as chest CT and X-ray images. These AI systems mainly differ in their choices of the algorithms for image segmentation, classification and disease diagnosis. Other AI-based systems have focused on predicting mortality rate, long-term patient hospitalization and patient outcomes for COVID-19. AI has huge potential in the battle against the COVID-19 pandemic but successful practical deployments of these AI-based tools have so far been limited due to challenges such as limited data accessibility, the need for external evaluation of AI models, the lack of awareness of AI experts of the regulatory landscape governing the deployment of AI tools in healthcare, the need for clinicians and other experts to work with AI experts in a multidisciplinary context and the need to address public concerns over data collection, privacy, and protection. Having a dedicated team with expertise in medical data collection, privacy, access and sharing, using federated learning whereby AI scientists hand over training algorithms to the healthcare institutions to train models locally, and taking full advantage of biomedical data stored in biobanks can alleviate some of problems posed by these challenges. Addressing these challenges will ultimately accelerate the translation of AI research into practical and useful solutions for combating pandemics.

show abstract

External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19

Cited by 57 publications

References 5 publications

On the role of artificial intelligence in medical imaging of COVID-19

On the role of artificial intelligence in medical imaging of COVID-19

On the Role of Artificial Intelligence in Medical Imaging of COVID-19

The Promise of AI in Detection, Diagnosis, and Epidemiology for Combating COVID-19: Beyond the Hype

Contact Info

Product

Resources

About