Machine Learning Models for Classifying High- and Low-Grade Gliomas: A Systematic Review and Quality of Reporting Analysis

Abstract

ObjectivesTo systematically review, assess the reporting quality of, and discuss improvement opportunities for studies describing machine learning (ML) models for glioma grade prediction.MethodsThis study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Diagnostic Test Accuracy (PRISMA-DTA) statement. A systematic search was performed in September 2020, and repeated in January 2021, on four databases: Embase, Medline, CENTRAL, and Web of Science Core Collection. Publications were screened in Covidence, and reporting quality was measured against the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. Descriptive statistics were calculated using GraphPad Prism 9.ResultsThe search identified 11,727 candidate articles with 1,135 articles undergoing full text review and 85 included in analysis. 67 (79%) articles were published between 2018-2021. The mean prediction accuracy of the best performing model in each study was 0.89 ± 0.09. The most common algorithm for conventional machine learning studies was Support Vector Machine (mean accuracy: 0.90 ± 0.07) and for deep learning studies was Convolutional Neural Network (mean accuracy: 0.91 ± 0.10). Only one study used both a large training dataset (n>200) and external validation (accuracy: 0.72) for their model. The mean adherence rate to TRIPOD was 44.5% ± 11.1%, with poor reporting adherence for model performance (0%), abstracts (0%), and titles (0%).ConclusionsThe application of ML to glioma grade prediction has grown substantially, with ML model studies reporting high predictive accuracies but lacking essential metrics and characteristics for assessing model performance. Several domains, including generalizability and reproducibility, warrant further attention to enable translation into clinical practice.Systematic Review RegistrationPROSPERO, identifier CRD42020209938.

Clinical implementation of artificial intelligence in neuroradiology with development of a novel workflow-efficient picture archiving and communication system-based automated brain tumor segmentation and radiomic feature extraction

“…Multiple works demonstrated high accuracy with use of ML for tumor segmentation, identification of images with brain tumors from other pathologies, glioma grade and molecular subtype prediction, differentiation of gliomas from lymphoma or brain metastases. These results suggest that clinical implementation of these algorithms is imminent and will be seen in the clinical practice in the next few years (Subramanian et al, 2021;Afridi et al, 2022;Avery et al, 2022;Bahar et al, 2022;Cassinelli Petersen et al, 2022;Jekel et al, 2022;Tillmanns et al, 2022). The next frontier in neuro-oncology imaging is identification of clinical applications of ML algorithms in clinical practice and determining the aspects of clinical care that can be improved with predictions that can be generated by these algorithms.…”

“…Future work will include implementation of advanced algorithms, such as nnUNET which show higher DSC scores. Currently machine learning in medical imaging is a hot topic with a large spike in publications starting in 2017 (Subramanian et al, 2021;Afridi et al, 2022;Avery et al, 2022;Bahar et al, 2022;Cassinelli Petersen et al, 2022;Jekel et al, 2022). Multiple works demonstrated high accuracy with use of ML for tumor segmentation, identification of images with brain tumors from other pathologies, glioma grade and molecular subtype prediction, differentiation of gliomas from lymphoma or brain metastases.…”

Machine Learning Tools for Image-Based Glioma Grading and the Quality of Their Reporting: Challenges and Opportunities

“…In our systematic review of 85 published ML studies developing models for image-based glioma grading, we found SVM and CNN to have mean accuracies of 90% and 91%, respectively [ 51 ]. Mean accuracies for these algorithms were similar across classification tasks regardless of whether the classification was binary or multi-class (e.g., 90% for the 24 studies whose best models performed binary classification of grades 1/2 vs. 3/4 compared to 86% for the 5 studies classifying grade 2 vs. 3 vs. 4).…”

“…Similar findings have been reported for glioma grade prediction literature. In our prior study conducting a TRIPOD analysis of more than 80 such model development studies, we report a mean adherence rate to TRIPOD of 44%, indicating poor quality of reporting [ 51 ]. Areas for improvement included reporting of titles and abstracts, justification of sample size, full model specification and performance, and participant demographics, and missing data.…”

“…Systematic reviews and meta-analyses in the field [ 41 , 51 , 64 ] reveal various aspects of reporting and bias risk that need to be addressed in order to promote complete understanding, rigorous assessment, and reproducibility of image-based ML glioma grading studies. Based on the problems identified in this literature (discussed in 4.4.2), we encourage future works to closely adhere to the reporting and risk of bias tools and guidelines most relevant to them, with particular attention to: Clearly signifying the development of a prediction model in their titles; Increasing the number of participants included in training/testing/validation sets; Justifying their choice of sample/sample size (whether that be on practical or logistical grounds) and approach to handling missing data (e.g., imputation); Specifying all components of model development (including data pre-processing and model calibration) and a full slate of performance metrics (accuracy, area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value, negative predictive value, and F1 score as well as associated confidence intervals) for training/testing/validation.…”