An illustration of reproducibility in neuroscience research in the absence of selective reporting

Abstract

The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multi-site collaborative study, we mapped brain anatomical left-right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we re-visited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and uncorrected significance at p<0.05. In this sense, the results within each dataset were viewed as coming from separate studies in an 'ideal publishing environment', i.e. free from selective reporting and p hacking. We found an average reproducibility rate per dataset, over all effects, of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. There is clearly substantial room to improve reproducibility in brain MRI research through increasing statistical power. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.

ENIGMA and global neuroscience: A decade of large-scale studies of the brain in health and disease across more than 40 countries

“…Thus, a valuable aspect of the existing ENIGMA studies is the ability to identify the most robust pattern of noninvasively measured neurobiological features involved in clinical syndromes across multiple samples that are more representative of the global population. This also results in robust effect size estimates, without the confounds of literature-based meta-analyses based on published data with possible publication bias (as noted in Kong et al) 18 . These data also provide a unique opportunity to assess important sources of disease heterogeneity, including key genetic, environmental, demographic, and psychosocial factors.…”

Does hippocampal volume explain performance differences on hippocampal-dependent tasks?

“…First, we had a sample of 217 participants. Samples of this size are much better powered 842 to find true and reliable relationships between brain structure and cognitive ability than the 843 smaller sample sizes (typically between 20 and 30 participants) often tested in previous studies 844 (Kharabian Masouleh et al, 2019; Kong & Francks, 2019). Indeed, the only other study to 845 employ a substantial sample was that of Weisberg et al (2019; n = 90) who also found no 846 relationship between hippocampal grey matter volume and navigation task performance.…”

Mapping brain asymmetry in health and disease through the ENIGMA consortium

“…Low statistical power in individual studies is also an important factor (Button, et al, 2013;Ioannidis, 2005). We carried out an empirical illustration of reproducibility in the absence of publication bias or p-hacking, by re-analyzing the summary statistics from our study of cerebral cortical asymmetry in 99 datasets (Kong, et al, 2019a;Kong, et al, 2018). For this purpose, we considered the meta-analytic hemispheric effect sizes (i.e., population-level asymmetry measures) to be 'true'.…”

“…Over the last four years, we have carried out studies of brain asymmetry in healthy individuals (Guadalupe, et al, 2017;Kong, et al, 2018) and individuals with disorders (de Kovel, et al, 2019;Kong, et al, 2019b;Postema, et al, 2019) using sample sizes roughly 1-2 orders of magnitude larger than previously achieved by the field. We also used summary statistics from the largest of these studies (based on data from over 17,000 participants) to address the critical issue of reproducibility in human neuroscience research (Kong, et al, 2019a). In this review, we summarize the general approach taken by our studies of brain asymmetry to date (de Kovel, et al, 2019;Guadalupe, et al, 2017;Kong, et al, 2019a;Kong, et al, 2019b;Kong, et al, 2018;Postema, et al, 2019), the most important findings and insights gained, as well as potential for future activities by the ENIGMA-Laterality Working Group.…”

“…We also used summary statistics from the largest of these studies (based on data from over 17,000 participants) to address the critical issue of reproducibility in human neuroscience research (Kong, et al, 2019a). In this review, we summarize the general approach taken by our studies of brain asymmetry to date (de Kovel, et al, 2019;Guadalupe, et al, 2017;Kong, et al, 2019a;Kong, et al, 2019b;Kong, et al, 2018;Postema, et al, 2019), the most important findings and insights gained, as well as potential for future activities by the ENIGMA-Laterality Working Group.…”