Reproducible brain-wide association studies require thousands of individuals

Abstract

Magnetic resonance imaging (MRI) has transformed our understanding of the human brain through well-replicated mapping of abilities to specific structures (for example, lesion studies) and functions1–3 (for example, task functional MRI (fMRI)). Mental health research and care have yet to realize similar advances from MRI. A primary challenge has been replicating associations between inter-individual differences in brain structure or function and complex cognitive or mental health phenotypes (brain-wide association studies (BWAS)). Such BWAS have typically relied on sample sizes appropriate for classical brain mapping4 (the median neuroimaging study sample size is about 25), but potentially too small for capturing reproducible brain–behavioural phenotype associations5,6. Here we used three of the largest neuroimaging datasets currently available—with a total sample size of around 50,000 individuals—to quantify BWAS effect sizes and reproducibility as a function of sample size. BWAS associations were smaller than previously thought, resulting in statistically underpowered studies, inflated effect sizes and replication failures at typical sample sizes. As sample sizes grew into the thousands, replication rates began to improve and effect size inflation decreased. More robust BWAS effects were detected for functional MRI (versus structural), cognitive tests (versus mental health questionnaires) and multivariate methods (versus univariate). Smaller than expected brain–phenotype associations and variability across population subsamples can explain widespread BWAS replication failures. In contrast to non-BWAS approaches with larger effects (for example, lesions, interventions and within-person), BWAS reproducibility requires samples with thousands of individuals.

Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study

“…Finally, although most behavioral measures were predicted better than chance, the prediction accuracies were low, especially for personality and mental health measures 121 . Further improvement will be necessary for clinical utility.…”

Brain functional connectivities that mediate the association between childhood traumatic events, and adult mental health and cognition

“…It is commonly observed that large effect sizes are associated with small sample sizes, and that the effect sizes generally become smaller, and better estimates of the real effect size, when the sample size is increased. 39 Part of the advantage of the large sample size utilized here is that it is likely to provide a robust estimate of the real effect size. It is further noted that it can be important in medical practice to know about effects even if they are small in size, for across a large population this could make a difference.…”

“…The maximum number of participants with data available were used, given the importance of large sample sizes in brain wide association analyses. 38 , 39 …”

Explainable Machine Learning Approach to Predict and Explain the Relationship between Task-based fMRI and Individual Differences in Cognition

“…However, this simplicity may come at a price. Recent findings have challenged the ability of the mass univariate approach in predicting individual differences (Kragel et al 2021; Marek et al 2022). For instance, in the context of resting-state fMRI and structural MRI, Marek and colleagues (2022), showed that mass univariate analyses had a much poorer ability in predicting individual differences in cognition, compared to multivariate analyses, or the techniques involving drawing information across different brain regions simultaneously in one model.…”

“…Recent findings have challenged the ability of the mass univariate approach in predicting individual differences (Kragel et al 2021; Marek et al 2022). For instance, in the context of resting-state fMRI and structural MRI, Marek and colleagues (2022), showed that mass univariate analyses had a much poorer ability in predicting individual differences in cognition, compared to multivariate analyses, or the techniques involving drawing information across different brain regions simultaneously in one model. Additionally, having separate tests for different brain regions in a mass-univariate fashion rests on the assumption that these regions are statistically independent of each other, which seems unrealistic given what is known about brain function.…”