Quality of reporting in AI cardiac MRI segmentation studies – A systematic review and recommendations for future studies

Alabed, Samer; Maiter, Ahmed; Salehi, Mahan; Mahmood, Aqeeb; Daniel, S; Jenkins, Sarah; Goodlad, Marcus; Sharkey, Michael; Mamalakis, Michail; Rakocevic, V; Dwivedi, Krit; Assadi, Hosamadin; Wild, Jim M.; Lu, Haiping; O’Regan, Declan P.; Geest, Rob J. van der; Garg, Pankaj; Swift, Andrew J.

doi:10.3389/fcvm.2022.956811

Cited by 17 publications

(12 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These two studies used publicly available Kaggle datasets for the detection of acute and chronic PE. Although the proportions of acute and chronic PE cases are available from the original data source, these were not stated by the studies themselves, limiting their transparency—it is best practice for publications to provide all relevant clinical characteristics regardless of whether they can be accessed elsewhere ( 17 , 18 ). Public datasets utilised in the studies may not have had all the elements or features required for an accurate PE diagnosis, which may limit the model's ability to identify the spectrum of abnormalities related to PE.…”

Section: Discussionmentioning

confidence: 99%

“…Extracted data included study information (such as location, year and journal type), study design, data selection (such as number of participants, number of CTEPH cases and inclusion criteria), and the AI model being presented (such as validation and performance results). The quality of each included study was appraised by checking compliance with the individual criteria of the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) ( 16 ), which were divided into four domains ( 17 , 18 ).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A systematic review of artificial intelligence tools for chronic pulmonary embolism on CT pulmonary angiography

Abdulaal,

Maiter,

Salehi

et al. 2024

Front. Radiol.

Self Cite

View full text Add to dashboard Cite

BackgroundChronic pulmonary embolism (PE) may result in pulmonary hypertension (CTEPH). Automated CT pulmonary angiography (CTPA) interpretation using artificial intelligence (AI) tools has the potential for improving diagnostic accuracy, reducing delays to diagnosis and yielding novel information of clinical value in CTEPH. This systematic review aimed to identify and appraise existing studies presenting AI tools for CTPA in the context of chronic PE and CTEPH.MethodsMEDLINE and EMBASE databases were searched on 11 September 2023. Journal publications presenting AI tools for CTPA in patients with chronic PE or CTEPH were eligible for inclusion. Information about model design, training and testing was extracted. Study quality was assessed using compliance with the Checklist for Artificial Intelligence in Medical Imaging (CLAIM).ResultsFive studies were eligible for inclusion, all of which presented deep learning AI models to evaluate PE. First study evaluated the lung parenchymal changes in chronic PE and two studies used an AI model to classify PE, with none directly assessing the pulmonary arteries. In addition, a separate study developed a CNN tool to distinguish chronic PE using 2D maximum intensity projection reconstructions. While another study assessed a novel automated approach to quantify hypoperfusion to help in the severity assessment of CTEPH. While descriptions of model design and training were reliable, descriptions of the datasets used in training and testing were more inconsistent.ConclusionIn contrast to AI tools for evaluation of acute PE, there has been limited investigation of AI-based approaches to characterising chronic PE and CTEPH on CTPA. Existing studies are limited by inconsistent reporting of the data used to train and test their models. This systematic review highlights an area of potential expansion for the field of AI in medical image interpretation.There is limited knowledge of A systematic review of artificial intelligence tools for chronic pulmonary embolism in CT. This systematic review provides an assessment on research that examined deep learning algorithms in detecting CTEPH on CTPA images, the number of studies assessing the utility of deep learning on CTPA in CTEPH was unclear and should be highlighted.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

A systematic review of artificial intelligence tools for chronic pulmonary embolism on CT pulmonary angiography

Abdulaal,

Maiter,

Salehi

et al. 2024

Front. Radiol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…High-quality reporting of model design, training, validation and testing is crucial for understanding the performance and generalisability of AI tools. 32 Transparency of AI tool development and performance is essential for the Open access gaining trust of stakeholders, including the public, and is therefore important for translation of tools into the clinical sphere. The way AI tools are presented should be consistent, enabling direct comparisons of performance, and accessible to all stakeholders, allowing purpose and performance to be understood by those without extensive experience in the field of AI.…”

Section: Discussionmentioning

confidence: 99%

Evaluating the performance of artificial intelligence software for lung nodule detection on chest radiographs in a retrospective real-world UK population

Maiter,

Hocking,

Matthews

et al. 2023

BMJ Open

Self Cite

View full text Add to dashboard Cite

ObjectivesEarly identification of lung cancer on chest radiographs improves patient outcomes. Artificial intelligence (AI) tools may increase diagnostic accuracy and streamline this pathway. This study evaluated the performance of commercially available AI-based software trained to identify cancerous lung nodules on chest radiographs.DesignThis retrospective study included primary care chest radiographs acquired in a UK centre. The software evaluated each radiograph independently and outputs were compared with two reference standards: (1) the radiologist report and (2) the diagnosis of cancer by multidisciplinary team decision. Failure analysis was performed by interrogating the software marker locations on radiographs.Participants5722 consecutive chest radiographs were included from 5592 patients (median age 59 years, 53.8% women, 1.6% prevalence of cancer).ResultsCompared with radiologist reports for nodule detection, the software demonstrated sensitivity 54.5% (95% CI 44.2% to 64.4%), specificity 83.2% (82.2% to 84.1%), positive predictive value (PPV) 5.5% (4.6% to 6.6%) and negative predictive value (NPV) 99.0% (98.8% to 99.2%). Compared with cancer diagnosis, the software demonstrated sensitivity 60.9% (50.1% to 70.9%), specificity 83.3% (82.3% to 84.2%), PPV 5.6% (4.8% to 6.6%) and NPV 99.2% (99.0% to 99.4%). Normal or variant anatomy was misidentified as an abnormality in 69.9% of the 943 false positive cases.ConclusionsThe software demonstrated considerable underperformance in this real-world patient cohort. Failure analysis suggested a lack of generalisability in the training and testing datasets as a potential factor. The low PPV carries the risk of over-investigation and limits the translation of the software to clinical practice. Our findings highlight the importance of training and testing software in representative datasets, with broader implications for the implementation of AI tools in imaging.

show abstract

“…The exclusion of semi-automated techniques, unpublished literature and conference abstracts were important to ensure consistent and reproducible evaluation of the included studies Recommendations for studies based on findings of this systematic review. Adapted from Alabed et al 2022 (13) and CLAIM (8). but did narrow the scope of the review and carried the risk of selection bias.…”

Section: The Systematic Reviewmentioning

confidence: 99%

How should studies using AI be reported? lessons from a systematic review in cardiac MRI

et al. 2023

Self Cite

View full text Add to dashboard Cite

Recent years have seen a dramatic increase in studies presenting artificial intelligence (AI) tools for cardiac imaging. Amongst these are AI tools that undertake segmentation of structures on cardiac MRI (CMR), an essential step in obtaining clinically relevant functional information. The quality of reporting of these studies carries significant implications for advancement of the field and the translation of AI tools to clinical practice. We recently undertook a systematic review to evaluate the quality of reporting of studies presenting automated approaches to segmentation in cardiac MRI (Alabed et al. 2022 Quality of reporting in AI cardiac MRI segmentation studies—a systematic review and recommendations for future studies. Frontiers in Cardiovascular Medicine 9:956811). 209 studies were assessed for compliance with the Checklist for AI in Medical Imaging (CLAIM), a framework for reporting. We found variable—and sometimes poor—quality of reporting and identified significant and frequently missing information in publications. Compliance with CLAIM was high for descriptions of models (100%, IQR 80%–100%), but lower than expected for descriptions of study design (71%, IQR 63–86%), datasets used in training and testing (63%, IQR 50%–67%) and model performance (60%, IQR 50%–70%). Here, we present a summary of our key findings, aimed at general readers who may not be experts in AI, and use them as a framework to discuss the factors determining quality of reporting, making recommendations for improving the reporting of research in this field. We aim to assist researchers in presenting their work and readers in their appraisal of evidence. Finally, we emphasise the need for close scrutiny of studies presenting AI tools, even in the face of the excitement surrounding AI in cardiac imaging.

show abstract

Quality of reporting in AI cardiac MRI segmentation studies – A systematic review and recommendations for future studies

Cited by 17 publications

References 14 publications

A systematic review of artificial intelligence tools for chronic pulmonary embolism on CT pulmonary angiography

A systematic review of artificial intelligence tools for chronic pulmonary embolism on CT pulmonary angiography

Evaluating the performance of artificial intelligence software for lung nodule detection on chest radiographs in a retrospective real-world UK population

How should studies using AI be reported? lessons from a systematic review in cardiac MRI

Contact Info

Product

Resources

About