Real-world evaluation of rapid and laboratory-free COVID-19 triage for emergency care: external validation and pilot deployment of artificial intelligence driven screening

Soltan, Andrew; Yang, Jenny; Pattanshetty, Ravi; Novak, Alex; Yang, Yang; Rohanian, Omid; Beer, Sally; Soltan, Marina; Thickett, David; Fairhead, Rory; Zhu, Tingting; Eyre, David W; Clifton, David A.; Watson, Adam; Bhargav, Akshay; Tough, Alex; Rogers, Alice; Shaikh, Ayisha; Valensise, Carolina; Lee, Charlotte; Otasowie, C; Metcalfe, David; Agarwal, Ekta; Zareh, Elham; Thangaraj, Evelyn; Pickles, Florence; Kelly, Gabriella; Tadikamalla, Gayatri; Shaw, George; Tong, Heather; Davies, Hettie; Bahra, Jasdeep; Morgan, James I.; Wilson, J. B.; Cutteridge, Joseph; O'Byrne, Katherine; Trajano, Luiza Farache; Oliver, Madeleine; Pikoula, Maria; Mendoza, Maya; Keevil, Melissa; Faisal, Muhammad; Dole, Natasha; Deal, Oscar; Conway-Jones, Rebecca; Sattar, Shajeel; Kundoor, Sneha; Shah, Sumaiyah; Muthusami, Vani

doi:10.1016/s2589-7500(21)00272-7

Cited by 40 publications

(94 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A balance must be found between the accuracy and the time needed to calculate the prediction that should be faster that qRT-PCR method. The future solution could potentially come from innovative strategies combining point-of-care testing and artificial intelligence-driven models as described recently by Soltan et al [ 41 ].…”

Section: Discussionmentioning

confidence: 99%

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

et al. 2022

View full text Add to dashboard Cite

Background Since the beginning of the pandemic, hospitals have been constantly overcrowded, with several observed waves of infected cases and hospitalisations. To avoid as much as possible this situation, efficient tools to facilitate the diagnosis of COVID-19 are needed. Objective To evaluate and compare prediction models to diagnose COVID-19 identified in a systematic review published recently using performance indicators such as discrimination and calibration measures. Methods A total of 1618 adult patients present at two Emergency Department triage centers and for whom qRT-PCR tests had been performed were included in this study. Six previously published models were reconstructed and assessed using diagnostic tests as sensitivity (Se) and negative predictive value (NPV), discrimination (Area Under the Roc Curve (AUROC)) and calibration measures. Agreement was also measured between them using Kappa’s coefficient and IntraClass Correlation Coefficient (ICC). A sensitivity analysis has been conducted by waves of patients. Results Among the 6 selected models, those based only on symptoms and/or risk exposure were found to be less efficient than those based on biological parameters and/or radiological examination with smallest AUROC values (< 0.80). However, all models showed good calibration and values above > 0.75 for Se and NPV but poor agreement (Kappa and ICC < 0.5) between them. The results of the first wave were similar to those of the second wave. Conclusion Although quite acceptable and similar results were found between all models, the importance of radiological examination was also emphasized, making it difficult to find an appropriate triage system to classify patients at risk for COVID-19.

show abstract

Section: Discussionmentioning

confidence: 99%

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

et al. 2022

View full text Add to dashboard Cite

show abstract

“…This sensitivity was chosen to exceed the sensitivity of lateral flow device (LFD) tests which are used in routine care. LFD sensitivity for OUH admissions between December 23, 2021 and March 6, 2021 was 56.9% (95% confidence interval 51.7–62.0%) 10 . Additionally, the gold standard for diagnosis of viral genome targets is by real-time polymerase chain reaction (RT-PCR), which has an estimated sensitivity of ~70% 27 .…”

Section: Methodsmentioning

confidence: 96%

“…Consistent with the training performed in ref. 10 , we used population median imputation to replace any missing values. We then standardized all features in our data to have a mean of 0 and a standard deviation of 1.…”

Section: Methodsmentioning

confidence: 99%

“…We aim to build on a previous study introduced in ref. 10 , where an ML pipeline (based on XGBoost) was used to screen patients attending hospital emergency departments for COVID-19. In this study, the authors trained and tested models using data from one NHS trust (Oxford University Hospitals; OUH), thereafter externally and prospectively validating the models across four independent trusts.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, using the same datasets as those used in ref. 10 , we developed and tested our models such that data from different sites are processed completely independent of the training data—as would be required for many external validation exercises in practice. To investigate a role for transfer learning, the current work uses a complex neural network approach which was previously shown to be effective at screening for COVID-19 11 , instead of the previously described XGBoost approach.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening

2022

Self Cite

View full text Add to dashboard Cite

As patient health information is highly regulated due to privacy concerns, most machine learning (ML)-based healthcare studies are unable to test on external patient cohorts, resulting in a gap between locally reported model performance and cross-site generalizability. Different approaches have been introduced for developing models across multiple clinical sites, however less attention has been given to adopting ready-made models in new settings. We introduce three methods to do this—(1) applying a ready-made model “as-is” (2); readjusting the decision threshold on the model’s output using site-specific data and (3); finetuning the model using site-specific data via transfer learning. Using a case study of COVID-19 diagnosis across four NHS Hospital Trusts, we show that all methods achieve clinically-effective performances (NPV > 0.959), with transfer learning achieving the best results (mean AUROCs between 0.870 and 0.925). Our models demonstrate that site-specific customization improves predictive performance when compared to other ready-made approaches.

show abstract

COVID-19 rapid molecular point-of-care testing is effective and cost-beneficial for the acute care of trauma patients

Stolberg-Stolberg

Jacob

Kuehn

et al. 2022

Eur J Trauma Emerg Surg

View full text Add to dashboard Cite

Purpose To evaluate the accuracy and cost benefit of a rapid molecular point-of-care testing (POCT) device detecting COVID-19 within a traumatological emergency department. Background Despite continuous withdrawal of COVID-19 restrictions, hospitals will remain particularly vulnerable to local outbreaks which is reflected by a higher institution-specific basic reproduction rate. Patients admitted to the emergency department with unknown COVID-19 infection status due to a- or oligosymptomatic COVID-19 infection put other patients and health care workers at risk, while fast diagnosis and treatment is necessary. Delayed testing results in additional costs to the health care system. Methods From the 8th of April 2021 until 31st of December 2021, all patients admitted to the emergency department were tested with routine RT-PCR and rapid molecular POCT device (Abbott ID NOW™ COVID-19). COVID-19-related additional costs for patients admitted via shock room or emergency department were calculated based on internal cost allocations. Results 1133 rapid molecular tests resulted in a sensitivity of 83.3% (95% CI 35.9–99.6%), specificity of 99.8% (95% CI 99.4–100%), a positive predictive value of 71.4% (95% CI 29–96.3%) and a negative predictive value of 99.9% (95% CI 99.5–100%) as compared to RT-PCR. Without rapid COVID-19 testing, each emergency department and shock room admission with subsequent surgery showed additional direct costs of 2631.25€, without surgery of 729.01€. Conclusion Although rapid molecular COVID-19 testing can initially be more expensive than RT-PCR, subsequent cost savings, improved workflows and workforce protection outweigh this effect by far. The data of this study support the use of a rapid molecular POCT device in a traumatological emergency department.

show abstract

Real-world evaluation of rapid and laboratory-free COVID-19 triage for emergency care: external validation and pilot deployment of artificial intelligence driven screening

Cited by 40 publications

References 30 publications

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening

COVID-19 rapid molecular point-of-care testing is effective and cost-beneficial for the acute care of trauma patients

Contact Info

Product

Resources

About