Sample size evolution in neuroimaging research: an evaluation of highly-cited studies (1990-2012) and of latest practices (2017-2018) in high-impact journals

Abstract

Timothy Myers for an initial check on sample sizes in about half the highly cited papers. We thank Josefína Weinerova for extracting sample size data for 2018. We thank Rik Henson (University of Cambridge) for comments on an earlier version of this manuscript.Author contributions. DS designed the research, extracted and analyzed data, wrote program code and the first draft of the paper. JPAI contributed critical comments on design, and data interpretation and revised successive drafts with DS.Competing financial interests. The authors declare no competing financial interests. Materials and Correspondence. Denes Szucs; ds377@cam.ac.ukWord count without References: 7780 words with tables and figure captions 2 Abstract, 191 wordsWe evaluated 1038 of the most cited structural and functional (fMRI) magnetic resonance brain imaging papers (1161 studies) published during 1990-2012 and 273 papers (302 studies) published in top neuroimaging journals in 2017 and 2018. 96% of highly cited experimental fMRI studies had a single group of participants and these studies had median sample size of 12, highly cited clinical fMRI studies (with patient participants) had median sample size of 14.5, and clinical structural MRI studies had median sample size of 50. The sample size of highly cited experimental fMRI studies increased at a rate of 0.74 participant/year and this rate of increase was commensurate with the median sample sizes of neuroimaging studies published in top neuroimaging journals in 2017 (23 participants) and 2018 (24 participants). Only 4 of 131 papers in 2017 and 5 of 142 papers in 2018 had prestudy power calculations, most for single t-tests and correlations. Only 14% of highly cited papers reported the number of excluded participants whereas about 45% of papers in 2017 and 2018 reported excluded participants. Targeted interventions from publishers and funders could facilitate increase in sample sizes and adherence to better standards.

Neuroimaging resilience to trauma: convergent evidence and challenges for future research

“…There are also some methodological weaknesses in some of the studies reviewed here. Much of the extant literature is based on small sample sizes, which are likely to be underpowered (Szucs and Ioannidis, 2020), and many older studies have also made use of overly-liberal multiple comparison correction methods, which may be subject to inflated type 1 error rates (Eklund et al, 2016;Cox et al, 2017). However, larger cohort-based functional imaging studies of resilience are starting to emerge, and some key findings have now been replicated in well-powered samples (e.g., Corral-Frías et al, 2015;Silveira et al, 2020).…”

Towards Reproducible Brain-Wide Association Studies

“…In Fig. 1e,f, we show that sampling variability (99% confidence interval [CI] of observed correlations) alone generates nominally significant (p<0.05), but inflated correlations, which would then be falsely reported 2,27 . We charted sampling variability as a function of sample size (N=25 to 3,928) for the strongest brain-wide associations as defined in the full sample (N=3,928, strict denoising).…”

“…Such brain-wide association studies (BWAS) hold great promise for predicting and reducing psychiatric disease burden and advancing our understanding of the cognitive abilities that underlie humanity's intellectual feats. However, obtaining MRI data remains very expensive (~$1,000/hr), resulting in many small-sample BWAS studies (e.g., median N=25 1,2 ), whose results often fail to replicate 1,[11][12][13][14][15] .…”

Can MRI measure myelin? Systematic review, qualitative assessment, and meta-analysis of studies validating microstructural imaging with myelin histology

“…To have a better overview of the literature, we provide quantitative summaries of our findings on the literature content ( Figure 2). We find that validation studies have a median sample size of 13 ( Figure 2A), comparable to the median of the most cited fMRI studies during the period of publication included in our meta-analysis (12), but below current median sample size (20) (Szucs and Ioannidis, 2020). The validation literature uses a wide range of histological markers (with 9 studies using more than one marker, Figure 2F).…”