Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center

Abstract

PurposeDespite the availability of commercial artificial intelligence (AI) tools for large vessel occlusion (LVO) detection, there is paucity of data comparing traditional machine learning and deep learning solutions in a real-world setting. The purpose of this study is to compare and validate the performance of two AI-based tools (RAPID LVO and CINA LVO) for LVO detection.Materials and methodsThis was a retrospective, single center study performed at a comprehensive stroke center from December 2020 to June 2021. CT angiography (n = 263) for suspected stroke were evaluated for LVO. RAPID LVO is a traditional machine learning model which primarily relies on vessel density threshold assessment, while CINA LVO is an end-to-end deep learning tool implemented with multiple neural networks for detection and localization tasks. Reasons for errors were also recorded.ResultsThere were 29 positive and 224 negative LVO cases by ground truth assessment. RAPID LVO demonstrated an accuracy of 0.86, sensitivity of 0.90, specificity of 0.86, positive predictive value of 0.45, and negative predictive value of 0.98, while CINA demonstrated an accuracy of 0.96, sensitivity of 0.76, specificity of 0.98, positive predictive value of 0.85, and negative predictive value of 0.97.ConclusionBoth tools successfully detected most anterior circulation occlusions. RAPID LVO had higher sensitivity while CINA LVO had higher accuracy and specificity. Interestingly, both tools were able to detect some, but not all M2 MCA occlusions. This is the first study to compare traditional and deep learning LVO tools in the clinical setting.

Evaluation of CINA® LVO artificial intelligence software for detection of large vessel occlusion in brain CT angiography

“…The high specificity of 93 % for the anterior circulation was similar to that shown in previous studies [10] , [11] , [12] . For the few observed false positives by the CINA® LVO software, a visible reason for the incorrect assessment could, in most cases, be deducted from the images included in the software’s report (for example two cases of partly thrombosed aneurysms and one case with a sharp M2 vessel curve marked as occlusions).…”

“…McClouth et al [10] found a sensitivity and specificity for overall LVO detection (including ICA, M1 and M2) to be 98 % and 98 %, respectively. In a third study, by Schlossman et al [12] , the analysis also indicated an uneven performance with a sensitivity of 55 % for LVO detection in the ICA and 87 % for M1 occlusions.…”

“…The CINA® LVO algorithm was trained on 566 LVO cases, all examined during 2018 and included from a single center in the United States of America [10] . Initial clinical evaluation of the CINA® LVO software in patients presenting with acute stroke symptoms has resulted in an overall sensitivity of 73–98 % and a specificity of 98–98 % in recent studies [10] , [11] , [12] . These kinds of metrics are however only valid within the specific study population, and therefore needs to be confirmed in other populations, including high-incidence populations.…”

Artificial Intelligence in Neuroradiology: A Review of Current Topics and Competition Challenges

“…Over the last few years, multiple studies have evaluated the performance of LVO detection for several available software packages with varying results. Sensitivities ranged from 73–98% and specificities ranged from 52–98% [ 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 ]. As a reference standard, in one study, neuroradiologists’ ability to detect anterior LVO using CTA and NCCT had sensitivities ranging from 75 to 88% and specificities ranging from 88 to 97% [ 37 ].…”

“…In contrast to other studies, posterior circulation and cervical ICA occlusions were included in the analysis. Just as it is for radiologists, AI-based algorithms have low sensitivity for detection of distal occlusions (i.e., M2 or M3) [ 31 , 33 , 35 ].…”