Dataset Size Sensitivity Analysis of Machine Learning Classifiers to Differentiate Molecular Markers of Pediatric Low-Grade Gliomas Based on MRI

Wagner, Matthias; Namdar, Khashayar; Alqabbani, Abdullah; Hainc, Nicolin; Figuereido, Liana Nobre; Sheng, Min; Shroff, Manohar; Bouffet, Éric; Tabori, Uri; Hawkins, Cynthia; Zhang, Michael; Yeom, Kristen W.; Khalvati, Farzad; Ertl‐Wagner, Birgit

doi:10.21203/rs.3.rs-883606/v1

Cited by 4 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When a ML algorithm is trained with a sufficiently large sample size, the dependence of the test accuracy on the test/train ratio should be small. 37 Figure 5e illustrates that the standard deviation in the test accuracy decreases monotonically from a sample size of 100−300 for all wavelengths in the ensemble model. This result indicates that while a sample size of 300 is optimal for this application, a sample size of 200 may be sufficient.…”

Section: ■ Results and Discussionmentioning

confidence: 97%

Using Machine Learning and Silver Nanoparticle-Based Surface-Enhanced Raman Spectroscopy for Classification of Cardiovascular Disease Biomarkers

Dixon,

Bonon,

Ivander

et al. 2023

ACS Appl. Nano Mater.

Self Cite

View full text Add to dashboard Cite

Characterizing complex biofluids using surface-enhanced Raman spectroscopy (SERS) coupled with machine learning (ML) has been proposed as a powerful tool for point-of-care detection of clinical disease. ML is well-suited to categorizing otherwise uninterpretable, patient-derived SERS spectra that contain a multitude of low concentration, disease-specific molecular biomarkers among a dense spectral background of biological molecules. However, ML can generate false, non-generalizable models when data sets used for model training are inadequate. It is thus critical to determine how different SERS experimental methodologies and workflow parameters can potentially impact ML disease classification of clinical samples. In this study, a labelfree, broadband, Ag nanoparticle-based SERS platform was coupled with ML to assess simulated clinical samples for cardiovascular disease (CVD), containing randomized combinations of five key CVD biomarkers at clinically relevant concentrations in serum.Raman spectra obtained at 532, 633, and 785 nm from up to 300 unique samples were classified into physiological and pathological categories using two standard ML models. Label-free SERS and ML could correctly classify randomized CVD samples with high accuracies of up to 90.0% at 532 nm using as few as 200 training samples. Spectra obtained at 532 nm produced the highest accuracies with no significant increase achieved using multiwavelength SERS. Sample preparation and measurement methodologies (e.g., different SERS substrate lots, sample volumes, sample sizes, and known variations in randomization and experimental handling) were shown to strongly influence the ML classification and could artificially increase classification accuracies by as much as 27%. This detailed investigation into the proper application of ML techniques for CVD classification can lead to improved data set acquisition required for the SERS community, such that ML on labeled and robust SERS data sets can be practically applied for future point-of-care testing in patients.

show abstract

Section: ■ Results and Discussionmentioning

confidence: 97%

Using Machine Learning and Silver Nanoparticle-Based Surface-Enhanced Raman Spectroscopy for Classification of Cardiovascular Disease Biomarkers

Dixon,

Bonon,

Ivander

et al. 2023

ACS Appl. Nano Mater.

Self Cite

View full text Add to dashboard Cite

show abstract

“…One study published in preprint, used neural networks to classify BRAF-mutational status in a single institution, though the algorithm required manual segmentation (16). The sensitivity of the dataset size on BRAF mutation classification performance was studied by Wagner et al in a radiomics based study (39). They showed that Neural networks outperform XGBoost for classification AUC and that the performance was affected by the size of the data used in training.…”

Section: Discussionmentioning

confidence: 99%

Noninvasive molecular subtyping of pediatric low-grade glioma with self-supervised transfer learning

Tak

Zapaishchykova

et al. 2023

Preprint

View full text Add to dashboard Cite

PurposeTo develop and externally validate a scan-to-prediction deep-learning pipeline for noninvasive, MRI-based BRAF mutational status classification for pLGG.Materials and MethodsWe conducted a retrospective study of two pLGG datasets with linked genomic and diagnostic T2-weighted MRI of patients: Boston Children’s Hospital (development dataset, N=214), and Child Brain Tumor Network (CBTN) (external validation, N=112). We developed a deep learning pipeline to classify BRAF mutational status (V600E vs. fusion vs. wild-type) from whole-scan input via a two-stage process: 1) 3D tumor segmentation and extraction of axial tumor images, and 2) slice-wise, deep learning-based classification of mutational status. We investigated knowledge-transfer approaches to prevent model overfitting with a primary endpoint of the area under the receiver operating characteristic curve (AUC). To enhance model interpretability, we developed a novel metric, COMDist that quantifies the accuracy of model attention with respect to the tumor.ResultsA combination of transfer learning from a pretrained medical imaging-specific network and self-supervised label cross-training (TransferX) coupled with consensus logic yielded the highest AUC, taken as a weighted average across the three mutational classes, (0.82 [95% CI: 0.70-0.90], Accuracy: 77%) on internal validation and (0.73 [95% CI 0.68-0.88], Accuracy: 75%) on external validation. Training with TransferX also led to an AUC improvement of 17.7% and a COMDist Improvement of 6.42% over training from scratch on the development dataset.ConclusionTransfer learning and self-supervised cross-training improved classification performance and generalizability for noninvasive pLGG mutational status prediction in a limited data scenario.Summary StatementThe authors developed and externally validated an automated, scan-to-prediction deep learning pipeline that classifies BRAF Mutational status in pediatric low-grade gliomas directly from T2-Weighted MRI scans.Key ResultsAn innovative training approach combining self-supervision and transfer learning (“TransferX”) is developed to boost model performance in low data settings;TransferX enables the development of a scan-to-prediction pipeline for pediatric LGG mutational status (BRAF V600E, fusion, or wildtype) with high accuracy and mild performance degradation on external validation;An evaluation metric “COMDist” is proposed to increase interpretability and quantify the accuracy of the model’s attention around the tumor.

show abstract

Improving Deep Learning Models for Pediatric Low-Grade Glioma Tumours Molecular Subtype Identification Using MRI-based 3D Probability Distributions of Tumour Location

Namdar,

Wagner,

Kudus

et al. 2024

Can Assoc Radiol J

View full text Add to dashboard Cite

Purpose: Pediatric low-grade gliomas (pLGG) are the most common brain tumour in children, and the molecular diagnosis of pLGG enables targeted treatment. We use MRI-based Convolutional Neural Networks (CNNs) for molecular subtype identification of pLGG and augment the models using tumour location probability maps. Materials and Methods: MRI FLAIR sequences of 214 patients (110 male, mean age of 8.54 years, 143 BRAF fused and 71 BRAF V600E mutated pLGG tumours) from January 2000 to December 2018 were included in this retrospective REB-approved study. Tumour segmentations (volumes of interest—VOIs) were provided by a pediatric neuroradiology fellow and verified by a pediatric neuroradiologist. Patients were randomly split into development and test sets with an 80/20 ratio. The 3D binary VOI masks for each class in the development set were combined to derive the probability density functions of tumour location. Three pipelines for molecular diagnosis of pLGG were developed: location-based, CNN-based, and hybrid. The experiment was repeated 100 times each with different model initializations and data splits, and the Areas Under the Receiver Operating Characteristic Curve (AUROC) was calculated, and Student’s t-test was conducted. Results: The location-based classifier achieved an AUROC of 77.9, 95% confidence interval (CI) (76.8, 79.0). CNN-based classifiers achieved an AUROC of 86.1, 95% CI (85.0, 87.3), while the tumour-location-guided CNNs outperformed the other classifiers with an average AUROC of 88.64, 95% CI (87.6, 89.7), which was statistically significant ( P-value .0018). Conclusion: Incorporating tumour location probability maps into CNN models led to significant improvements for molecular subtype identification of pLGG.

show abstract

Dataset Size Sensitivity Analysis of Machine Learning Classifiers to Differentiate Molecular Markers of Pediatric Low-Grade Gliomas Based on MRI

Cited by 4 publications

References 27 publications

Using Machine Learning and Silver Nanoparticle-Based Surface-Enhanced Raman Spectroscopy for Classification of Cardiovascular Disease Biomarkers

Using Machine Learning and Silver Nanoparticle-Based Surface-Enhanced Raman Spectroscopy for Classification of Cardiovascular Disease Biomarkers

Noninvasive molecular subtyping of pediatric low-grade glioma with self-supervised transfer learning

Improving Deep Learning Models for Pediatric Low-Grade Glioma Tumours Molecular Subtype Identification Using MRI-based 3D Probability Distributions of Tumour Location

Contact Info

Product

Resources

About