Small effect size leads to reproducibility failure in resting-state fMRI studies

Abstract

Thousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a between-group design and 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the effect size of brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric disorders together with healthy individuals have been analyzed. Parkinson's disease off levodopa (PD-off) included 687 participants from 15 studies. PD on levodopa (PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named amplitude of low frequency fluctuation (ALFF) showed that the effect size (Hedges' g) was 0.19 -0.39 for the 4 datasets using between-group design and 0.46 for the dataset using within-group design. The effect size of PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the between-group design results of each study showed high false negative rates (median 99%), high false discovery rates (median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.

Characterizing the impact of adversity, abuse, and neglect on adolescent amygdala resting-state functional connectivity

“…Finally, we employed a single analysis pipeline that was pre-registered to protect against pipeline exploration that would bias us toward positive results. However, variation in analytic processing streams impacts findings and/or data quality in task-based ( Botvinik-Nezer et al, 2020 ) and resting-state ( Ciric et al, 2016 ) functional neuroimaging, and individual studies of between-group rs-fc differences using single analytic approaches may be prone to error ( Jia et al, 2018 ). These same meta-science studies find that meta-analyses examining unthresholded statistical maps across processing streams and replication samples reveal patterns that are robust.…”

Higher Sensitivity and Reproducibility of Wavelet-Based Amplitude of Resting-State fMRI

“…Two-sample t-tests between ADHD group and TDC group were performed in each cohort for ADHD-200 dataset. As Jia et al (2018) have recently reported, stringent or liberal multiple comparison correction could not control the false discoveries across multiple studies when the effect sizes were relatively small. The reproducibility of the results across multiple cohorts is more important for the recovery of the ground truth.…”

Interactions between monoamine oxidase A rs1137070 and smoking on brain structure and function in male smokers

“…How to balance the type I and type II error is a big issue in the field. Recently, a study (Jia et al ., ) used the meta‐analysis results as the robust results and found that the between‐group design results of each original study showed high‐false negative rates (median 99%), high‐false discovery rates (median 86%) and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small.…”