Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis

Abstract

There have been few independent evaluations of computer-aided detection (CAD) software for tuberculosis (TB) screening, despite the rapidly expanding array of available CAD solutions. We developed a test library of chest X-ray (CXR) images which was blindly re-read by two TB clinicians with different levels of experience and then processed by 12 CAD software solutions. Using Xpert MTB/RIF results as the reference standard, we compared the performance characteristics of each CAD software against both an Expert and Intermediate Reader, using cut-off thresholds which were selected to match the sensitivity of each human reader. Six CAD systems performed on par with the Expert Reader (Qure.ai, DeepTek, Delft Imaging, JF Healthcare, OXIPIT, and Lunit) and one additional software (Infervision) performed on par with the Intermediate Reader only. Qure.ai, Delft Imaging and Lunit were the only software to perform significantly better than the Intermediate Reader. The majority of these CAD software showed significantly lower performance among participants with a past history of TB. The radiography equipment used to capture the CXR image was also shown to affect performance for some CAD software. TB program implementers now have a wide selection of quality CAD software solutions to utilize in their CXR screening initiatives.

Diagnostic accuracy of three computer-aided detection systems for detecting pulmonary tuberculosis on chest radiography when used for screening: analysis of an international, multicenter migrants screening study

“…Subgroup analyses showed that the performance of CADs can vary among some population demographic and clinical characteristics. All CAD systems performed worse in participants with a history of TB, something which has also been observed in previous studies [13–15]. This is to be expected, as healed TB can leave residual CXR changes, which usually are classified as TB findings on CXR but can lead to negative microbiological test results.…”

“…Moreover, most studies used non-expert CXR interpreters and assessed an online CAD processing system or shared images with the CAD vendors and compared the performance against a suboptimal refence standard of a single sputum specimen tested with Xpert MTB/RIF which further highlights the need for independent and rigorous studies [8–12]. More recent investigations have focused on offline and multiple AI systems [13–15], but they remain few in number.…”

Reimagining the status quo: How close are we to rapid sputum-free tuberculosis diagnostics for all?

“…48 An evaluation of 1032 images demonstrated that six out of 12 CAD platforms (Qure.ai, DeepTek, Delft Imaging, JF Healthcare, OXIPIT, Lunit) performed similarly to an expert reader, only three of which (Qure.ai, Delft Imaging and Lunit) performed significantly better than an intermediate reader. 49 A large evaluation of almost 24 000 outpatients, the majority of whom had symptoms, demonstrated that all five of the algorithms evaluated reduced the number of molecular tests required by 50% while maintaining an overall sensitivity of 90%. Two products: qXR (qure.ai, India) and CAD4TB (Delft Imaging Systems, Netherlands) met the triage TPP criteria.…”

World TB Day 2022: Revamping and Reshaping Global TB Control Programs by Advancing Lessons learnt from the COVID-19 pandemic

“…Thus these lessons learnt from several COVID-19 innovations in the health systems delivery during lockdowns, and the rapid development and rollout of diagnostics and vaccines highlight the need to stimulate ambitious political and scientific actions for revamping global TB control efforts whilst COVID-19 is brought under control (Ntoumi F et al, 2022a;2022b;Chakaya et al, 2022-IJID;Pai et al, 2022;Zimmer AJ et al, 2021;Ruhwald M et al, 2021Ruhwald M et al, , 2022Hopewell PC et al, 2020;Chapman H et al, 2021;Sahu S et al, 2021: Keene C et al, 2020. This should also include use of Artificial Intelligence (AI) for improved TB screening at all points of care and rapid data communication (Codlin et al, 2021: Malik et al, 2021. Obtaining accurate data on the actual global burden of LTBI, TB, DRTB, and TB-related deaths is essential to strengthen the evidence base required to convince media, governments and donors to pay specific attention to TB and its continuing status as a global public health emergency.…”