Towards Reproducible Brain-Wide Association Studies

Abstract

Magnetic resonance imaging (MRI) continues to drive many important neuroscientific advances. However, progress in uncovering reproducible associations between individual differences in brain structure/function and behavioral phenotypes (e.g., cognition, mental health) may have been undermined by typical neuroimaging sample sizes (median N=25)1,2. Leveraging the Adolescent Brain Cognitive Development (ABCD) Study3 (N=11,878), we estimated the effect sizes and reproducibility of these brain wide associations studies (BWAS) as a function of sample size. The very largest, replicable brain wide associations for univariate and multivariate methods were r=0.14 and r=0.34, respectively. In smaller samples, typical for brain wide association studies, irreproducible, inflated effect sizes were ubiquitous, no matter the method (univariate, multivariate). Until sample sizes started to approach consortium levels, BWAS were underpowered and statistical errors assured. Multiple factors contribute to replication failures4,5,6; here, we show that the pairing of small brain behavioral phenotype effect sizes with sampling variability is a key element in widespread BWAS replication failure. Brain behavioral phenotype associations stabilize and become more reproducible with sample sizes of N>2,000. While investigator initiated brain behavior research continues to generate hypotheses and propel innovation, large consortia are needed to usher in a new era of reproducible human brain wide association studies.

A Large-Scale Structural and Functional Connectome of Social Mentalizing

“…Future research needs to develop proper behavioral paradigms (e.g. using more ecologically valid social tasks) [137,138] to map individual ToM skills to connectome features and also test their reproducibility across multiple datasets [139].…”

Behavioral and brain signatures of substance use vulnerability in childhood

“…Smith & Nichols, 2018)-including within the ABCD Study (Marek et al, 2020)-and conversely, that statistically significant effects in underpowered studies are likely to be inflated (Button et al, 2013;Ioannidis, 2008;Marek et al, 2020;Yarkoni, 2009), it is unsurprising that the effect sizes observed in the current study are small (partial correlation [rp] ~= 0.1). However, a small effect in a large sample size potentially allows for higher replicability (Marek et al, 2020;Scheinost et al, 2019), which is further demonstrated here via cross-validation procedures and testing on novel participants. Finally, it is worth noting that the effects reported here are observed within children…”

“…Future work will be needed to assess the robustness of this data reduction step relative to other techniques such as polychoric correlation, which may be better suited for variables of mixed types (i.e., ordinal, numerical) (Holgado-Tello & Chacón-Moscoso, 2010). Moreover, the number of participants lost to motion were substantial-in part due to rigorous motion thresholds often applied to adult data (Horien et al, 2018;Kumar et al, 2019;Rosenberg et al, 2016a;Rosenberg et al, 2016b), which is similarly observed by others using rigorous quality control procedures within ABCD (Marek et al, 2020;Sripada et al, 2019) and in connectivity-based studies in adults and adolescents (Greene et al, 2018;Horien et al, 2019). Analytic tools such as censoring (Power et al, 2014) may prevent data loss due to motion, which may subsequently allow for greater variability in the distribution of behavioral variables of potential interest that may be related to motion (e.g., impulsivity).…”

Cognitive insights from evolutionarily new brain structures in prefrontal cortex

“…Additionally, while there is "no one-size-fits-all sample size for neuroimaging studies" 84 and we had a large N (>1000) in terms of sulci explored in the present study, new methods and tools will need to be developed to increase the number of subjects in futures studies. Ongoing work is already underway to develop deep learning algorithms to accurately define tertiary sulci automatically in individual subjects, and initial results are promising.…”