Multisite generalizability of schizophrenia diagnosis classification based on functional brain connectivity

Orban, Pierre; Christian, Dansereau; Desbois, Laurence; Mongeau‐Pérusse, Violaine; Giguère, Charles–Édouard; Nguyen, Hien D.; Mendrek, Adrianna; Stip, Émmanuel; Bellec, Pierre

doi:10.1101/141192

Cited by 6 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Again, we obtained low average accuracies (with means in the range 0.51 -0.66), reinforcing the evidence for disease heterogeneity. Interestingly, these results are consistent with a recent study [Orban et al, 2017] on multisite generalizability of schizophrenia diagnosis based on functional brain connectivity, which reported multisite classification accuracies below 70%, in contrast to over 30 previously published, largely single-site schizophrenia studies, whose average reported classification accuracy exceeds 80%.…”

Section: Discussionsupporting

confidence: 91%

Exploring the reproducibility of functional connectivity alterations in Parkinson’s disease

Badea

Onu

et al. 2017

PLoS ONE

View full text Add to dashboard Cite

Since anatomic MRI is presently not able to directly discern neuronal loss in Parkinson’s Disease (PD), studying the associated functional connectivity (FC) changes seems a promising approach toward developing non-invasive and non-radioactive neuroimaging markers for this disease. While several groups have reported such FC changes in PD, there are also significant discrepancies between studies. Investigating the reproducibility of PD-related FC changes on independent datasets is therefore of crucial importance. We acquired resting-state fMRI scans for 43 subjects (27 patients and 16 normal controls, with 2 replicate scans per subject) and compared the observed FC changes with those obtained in two independent datasets, one made available by the PPMI consortium (91 patients, 18 controls) and a second one by the group of Tao Wu (20 patients, 20 controls). Unfortunately, PD-related functional connectivity changes turned out to be non-reproducible across datasets. This could be due to disease heterogeneity, but also to technical differences. To distinguish between the two, we devised a method to directly check for disease heterogeneity using random splits of a single dataset. Since we still observe non-reproducibility in a large fraction of random splits of the same dataset, we conclude that functional heterogeneity may be a dominating factor behind the lack of reproducibility of FC alterations in different rs-fMRI studies of PD. While global PD-related functional connectivity changes were non-reproducible across datasets, we identified a few individual brain region pairs with marginally consistent FC changes across all three datasets. However, training classifiers on each one of the three datasets to discriminate PD scans from controls produced only low accuracies on the remaining two test datasets. Moreover, classifiers trained and tested on random splits of the same dataset (which are technically homogeneous) also had low test accuracies, directly substantiating disease heterogeneity.

show abstract

Section: Discussionsupporting

confidence: 91%

Exploring the reproducibility of functional connectivity alterations in Parkinson’s disease

Badea

Onu

et al. 2017

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…The debate about the complex composition of global signals and the necessity of GSR in data preprocessing and data analysis always exists in most cases [24,54,55]. Some studies reported that the global signal was likely to reflect important neuronal components in rs-fMRI data [17,18,39]. Qing et al [26] reported that GSR effects are region-specific and suggested that it is great to report results both with and without GSR in ReHo study.…”

Section: Disease Markersmentioning

confidence: 99%

“…An increasing number of studies have reported that multimodal brain data can improve diagnostic accuracy by combining the information obtained from different MRI imaging modalities [8,[14][15][16]. The machine learning (ML) technique is a new approach that can extract relevant information from images and construct models to determine the probability of disease onset, and it can make a higher accurate prediction compared with conventional methods [5,6,13,17,18]. Salvador et al [15] achieved 75.76% accuracy in schizophrenia diagnosis, and de Filippis et al [5] reported that support vector machine associated with other ML techniques could achieve accuracy close to 100%.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning of Schizophrenia Detection with Structural and Functional Neuroimaging

Shi

Zhang

et al. 2021

Disease Markers

View full text Add to dashboard Cite

Schizophrenia (SZ) is a severe psychiatric illness, and it affects around 1% of the general population; however, its reliable diagnosis is challenging. Functional MRI (fMRI) and structural MRI (sMRI) are useful techniques for investigating the functional and structural abnormalities of the human brain, and a growing number of studies have reported that multimodal brain data can improve diagnostic accuracy. Machine learning (ML) is widely used in the diagnosis of neuroscience and neuropsychiatry diseases, and it can obtain high accuracy. However, the conventional ML which concatenated the features into a longer feature vector could not be sufficiently effective to combine different features from different modalities. There are considerable controversies over the use of global signal regression (GSR), and few studies have explored the role of GSR in ML in diagnosing neurological diseases. The current study utilized fMRI and sMRI data to implement a new method named multimodal imaging and multilevel characterization with multiclassifier (M3) to classify SZs and healthy controls (HCs) and investigate the influence of GSR in SZ classification. We found that when we used Brainnetome 246 atlas and without performed GSR, our method obtained a classification accuracy of 83.49%, with a sensitivity of 68.69%, a specificity of 93.75%, and an AUC of 0.8491, respectively. We also got great classification performances with different processing methods (with/without GSR and different brain parcellation schemes). We found that the accuracy and specificity of the models without GSR were higher than that of the models with GSR. Our findings indicate that the M3 method is an effective tool to distinguish SZs from HCs, and it can identify discriminative regions to detect SZ to explore the neural mechanisms underlying SZ. The global signal may contain important neuronal information; it can improve the accuracy and specificity of SZ detection.

show abstract

“…Bigger data sets are needed to avoid overfitting and to build a classifier with better generalizability. Second, only one modality of data (rsfMRI) was utilized, even though both functional and structural brain information may be important for high-accuracy classification using machine learning (Mikolas et al, 2017;Orban et al, 2018;Ota et al, 2012). Third, we did not collect the smoking status of our participants in this study.…”

Section: Limitationmentioning

confidence: 99%

Generalizability of machine learning for classification of schizophrenia based on resting‐state functional MRI data

Cai

Xie

Madsen

et al. 2019

Human Brain Mapping

View full text Add to dashboard Cite

Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within‐site and between‐site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting‐state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within‐site generalizability of the classification framework in the main data set using cross‐validation. Then, we trained a model in the main data set and investigated between‐site generalization in the validated data set using external validation. Finally, recognizing the poor between‐site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between‐site classification performance. Cross‐validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within‐site cross‐validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.

show abstract

Multisite generalizability of schizophrenia diagnosis classification based on functional brain connectivity

Cited by 6 publications

References 21 publications

Exploring the reproducibility of functional connectivity alterations in Parkinson’s disease

Exploring the reproducibility of functional connectivity alterations in Parkinson’s disease

Machine Learning of Schizophrenia Detection with Structural and Functional Neuroimaging

Generalizability of machine learning for classification of schizophrenia based on resting‐state functional MRI data

Contact Info

Product

Resources

About