Multisite generalizability of schizophrenia diagnosis classification based on functional brain connectivity

Orban, Pierre; Dansereau, Christian; Desbois, Laurence; Mongeau‐Pérusse, Violaine; Giguère, Charles–Édouard; Nguyen, Hien D.; Mendrek, Adrianna; Stip, Emmanuel; Bellec, Pierre

doi:10.1016/j.schres.2017.05.027

Cited by 50 publications

(39 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This bias might even be modality-specific, in that non-brain features might generalize better than brain imaging features (Bhagwat et al, 2018). Training predictive models on data from multiple sites has been shown to improve generalization (Abraham et al, 2017; Liem et al, 2017; Orban et al, 2018). Hence, future studies should use models trained and tested on data from multiple sites, which requires further suitable longitudinal and publicly available datasets (Varoquaux, 2018).…”

Section: Discussionmentioning

confidence: 99%

Predicting future cognitive decline from non-brain and multimodal brain imaging data in healthy and pathological aging

Liem

Dadi

Engemann

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Cognitive decline occurs in healthy and pathological aging, and both may be preceded by subtle changes in the brain — offering a basis for cognitive predictions. Previous work has largely focused on predicting a diagnostic label from structural brain imaging. Our study broadens the scope of applications to cognitive decline in healthy aging by predicting future decline as a continuous trajectory, rather than a diagnostic label. Furthermore, since brain structure as well as function changes in aging, it is reasonable to expect predictive gains when using multiple brain imaging modalities. Here, we tested whether baseline multimodal neuroimaging data improve the prediction of future cognitive decline in healthy and pathological aging. Non-brain data (including demographics and clinical and neuropsychological scores) were combined with structural and functional connectivity MRI data from the OASIS-3 project (N = 662; age = 46 – 96y). The combined input data was entered into cross-validated multi-target random forest models to predict future cognitive decline (measured by the Clinical Dementia Rating and the Mini-Mental State Examination), on average 5.8y into the future. The analysis was preregistered and all analysis code is publicly available. We found that combining non-brain with structural data improved the continuous prediction of future cognitive decline (best test-set performance: R2 = 0.42) and that cognitive performance, daily functioning, and subcortical volume drove the performance of our model. In contrast, including functional connectivity did not improve predictive accuracy. In the future, the prognosis of age-related cognitive decline may enable earlier and more effective cognitive, pharmacological, and behavioral interventions to be tailored to the individual.

show abstract

Section: Discussionmentioning

confidence: 99%

Predicting future cognitive decline from non-brain and multimodal brain imaging data in healthy and pathological aging

Liem

Dadi

Engemann

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Bigger data sets are needed to avoid overfitting and to build a classifier with better generalizability. Second, only one modality of data (rsfMRI) was utilized, even though both functional and structural brain information may be important for high‐accuracy classification using machine learning (Mikolas et al, ; Orban et al, ; Ota et al, ). Third, we did not collect the smoking status of our participants in this study.…”

Section: Discussionmentioning

confidence: 99%

Generalizability of machine learning for classification of schizophrenia based on resting‐state functional MRI data

Cai

Xie

Madsen

et al. 2019

Human Brain Mapping

View full text Add to dashboard Cite

Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within‐site and between‐site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting‐state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within‐site generalizability of the classification framework in the main data set using cross‐validation. Then, we trained a model in the main data set and investigated between‐site generalization in the validated data set using external validation. Finally, recognizing the poor between‐site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between‐site classification performance. Cross‐validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within‐site cross‐validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.

show abstract

“…It may thus be possible to extract multivariate biomarkers of brain diseases from such multisite harmonized data, as planned in the CCNA. Recent papers (Abraham et al, 2016;Orban et al, 2017) also showed that machine learning models trained on multisite rsfMRI data generalize better to subjects from new unseen sites, than models trained on single site data. Better approaches for site harmonization, either prospective like CDIP, or retrospective, for example (Yan et al, 2013) , may still increase the precision of rsfMRI biomarkers, and is an important area of future work.…”

Section: Precision Medicinementioning

confidence: 99%

Multivariate consistency of resting-state fMRI connectivity maps acquired on a single individual over 2.5 years, 13 sites and 3 vendors

Badhwar

Collin-Verreault

Orban

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Running title: Multivariate consistency of long-term multisite resting-state fMRI CDIP rsfMRI manuscript Badhwar et al. AbstractStudies using resting-state functional magnetic resonance imaging (rsfMRI) are increasingly collecting data at multiple sites in order to speed up recruitment or increase sample size. The main objective of this study was to assess the long-term consistency of rsfMRI connectivity maps derived at multiple sites and vendors using the Canadian Dementia Imaging Protocol (CDIP, www.cdip-pcid.ca ). Nine to ten minutes of functional BOLD images were acquired from an adult cognitively healthy volunteer scanned repeatedly at 13 Canadian sites on three scanner makes (General Electric, Philips and Siemens) over the course of 2.5 years. The consistency (spatial Pearson's correlation) of rsfMRI connectivity maps for seven canonical networks ranged from about 0.4-0.8 (intra-site) to 0.3-0.8 (inter-vendor), with a negligible effect of time. We noted systematic differences in data quality across vendors, which may also explain some of these results. We also pooled the long-term longitudinal data with a single-site, short-term (1 month) data sample acquired on 26 subjects (10 scans per subject), called HNU1. Using randomly selected pairs of scans from each subject, we quantified the ability of a data-driven unsupervised cluster analysis to match two scans of the same subjects. In this "fingerprinting" experiment, we found that scans from the Canadian subject (Csub) could be matched with high accuracy intra-site (>95% for some networks), but that the accuracy decreased substantially for scans drawn from different sites and vendors, while still remaining in the range of accuracies observed in HNU1. Overall, our results demonstrate good multivariate stability of rsfMRI measures over several years, but substantial impact of scanning site and vendors. How detrimental these effects are will depend on the application, yet improving methods for harmonizing multisite analysis is an important area for future work.

show abstract

Multisite generalizability of schizophrenia diagnosis classification based on functional brain connectivity

Cited by 50 publications

References 22 publications

Predicting future cognitive decline from non-brain and multimodal brain imaging data in healthy and pathological aging

Predicting future cognitive decline from non-brain and multimodal brain imaging data in healthy and pathological aging

Generalizability of machine learning for classification of schizophrenia based on resting‐state functional MRI data

Multivariate consistency of resting-state fMRI connectivity maps acquired on a single individual over 2.5 years, 13 sites and 3 vendors

Contact Info

Product

Resources

About