Improving the replicability of neuroimaging findings by thresholding effect sizes instead of <scp><i>p</i></scp>‐values

Vandekar, Simon; Stephens, Jeremy

doi:10.1002/hbm.25374

Cited by 9 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The problems of thresholding p ‐values and other test statistics have been addressed at length in the current statistics literature (e.g., Wasserstein et al, 2019), as well as from the specific perspective of fMRI by Chen et al (2022). Vandekar and Stephens (2021), by contrast, suggest that thresholding effect sizes rather than p ‐values can lead to increased replicability. Several other recent works have also proposed approaches for replicable research specifically in the context of large‐sample fMRI studies (Abraham et al, 2017; Abrol et al, 2017).…”

Section: Reproducibility and Replicabilitymentioning

confidence: 99%

Functional neuroimaging in the era of Big Data and Open Science: A modern overview

Lazar

2023

WIREs Computational Stats

View full text Add to dashboard Cite

In the past 30 years, the statistical analysis of functional neuroimaging data has made much progress, and spurred many new research directions. At the same time, problems with reproducibility and replicability have plagued the field, owing in part to small sample sizes, a plethora of choices at the data preprocessing stage, and overall lack of transparency in reporting. The latter two in particular pose barriers to statisticians who want to become involved in the area. Recent efforts by some in the neuroimaging community to address these problems represent a turning point. This article highlights the current landscape and provides an introduction to some of the relevant resources in “open neuroimaging.”This article is categorized under: Software for Computational Statistics > Software/Statistical Software

show abstract

Section: Reproducibility and Replicabilitymentioning

confidence: 99%

Functional neuroimaging in the era of Big Data and Open Science: A modern overview

Lazar

2023

WIREs Computational Stats

View full text Add to dashboard Cite

show abstract

“…This criterion is chosen for various reasons. First, using effect size as a threshold has been shown to improve replicability in neuroimaging findings (Vandekar and Stephens, 2021). Second, a recent study found that effect sizes of association between two variables in ABCD were mostly around 0.03 to 0.09 (Owens et al, 2021).…”

Section: Measurementsmentioning

confidence: 99%

Long-lasting effects of insufficient sleep on neurocognitive development in early adolescence

Yang

Wang

2022

Preprint

View full text Add to dashboard Cite

Importance: Adolescents nowadays often get insufficient sleep. Yet, the long-term adverse effects of sleep loss on developing brain and behavior remains unknown. Objective: To determine whether insufficient sleep leads to long-lasting impacts on mental health, cognition, and brain development in adolescents across two years. Design: This longitudinal study utilized a public dataset, the Adolescent Brain Cognitive Development (ABCD) study, which is an ongoing study starting from 2016. Setting: Data were collected from 21 research sites in the U.S. Participants: 11,875 9-10-year-olds were recruited using stratified sampling in order to reflect the diversity of the U.S. population. Intervention: Individuals with sufficient versus insufficient sleep (< 9 hours per day for adolescents) were compared after controlling for age (months), sex, race, puberty status, and other 7 covariates based on propensity score matching. Main Outcomes and Measures: Behavior problems, cognition, mental health assessments, resting-state functional connectivity, gray matter volume, cortical area, cortical thickness, and structural connectivity (Fractional anisotropy) were collected and preprocessed by the ABCD study. Independent-sample t-tests and meditation analysis were performed to investigate the effects of insufficient sleep. Results: 3021 matched pairs (50.7% male) were identified based on baseline assessment, with mean (SD) age of 119.5 (7.5) months. In baseline, sufficient sleep is associated less behavioral problems on 18 of 20 assessments, e.g. depress (95% CI of mean difference: -0.28 to -0.47, false discovery rate (FDR)-corrected p < .001, Cohen's d = -0.20), better cognitive performance on 7 of 10 assessments, such as crystal cognition (95% CI: 0.81 to 1.50, FDR-corrected p < .001, Cohen's d = 0.17), better functional connection between cortical regions and basal ganglia (all FDR-corrected p < .05, Cohen's d >0.15), and large structure in ACC and temporal pole (all FDR-corrected p < .05, Cohen's d >0.09). Similar patterns of effect of sufficient sleep were found in FL2 (749 pairs remained) e.g. Cohen's d of function connectivity at baseline was correlated with Cohen's d of that at FL2 (r = 0.54, 95% CI: 0.45 to 0.61, p < 1e-10). Mediation and longitudinal mediation analysis revealed that identified brain measures (e.g. gray matter volume of left temporal pole) at baseline mediated the effect of sufficient sleep on behavioral assessments (e.g. crystal cognition) at baseline and at FL2 (95% CI did not encompass 0, p < 0.05 on 100,000 random-generated bootstrapped samples). Conclusions and Relevance: These results provide strong population-level evidence for the long-lasting detrimental effects of insufficient sleep on mental health, cognition, and brain function and structure in adolescents. The current study identified potential neural mechanisms of adverse effect of insufficient sleep in adolescents, which might provide a theoretical grounding for sleep intervention programs to improve the long-term developmental outcomes in adolescents.

show abstract

Improving the replicability of neuroimaging findings by thresholding effect sizes instead of p‐values

Vandekar

Stephens

2021

Human Brain Mapping

Self Cite

View full text Add to dashboard Cite

The classical approach for testing statistical images using spatial extent inference (SEI) thresholds the statistical image based on the p-value. This approach has an unfortunate consequence on the replicability of neuroimaging findings because the targeted brain regions are affected by the sample size-larger studies have more power to detect smaller effects. Here, we use simulations based on the preprocessed Autism Brain Imaging Data Exchange (ABIDE) to show that thresholding statistical images by effect sizes has more consistent estimates of activated regions across studies than thresholding by p-values. Using a constant effect size threshold means that the p-value threshold naturally scales with the sample size to ensure that the target set is similar across repetitions of the study that use different sample sizes. As a consequence of thresholding by the effect size, the type 1 and type 2 error rates go to zero as the sample size gets larger. We use a newly proposed robust effect size index that is defined for an arbitrary statistical image so that effect size thresholding can be used regardless of the test statistic or model.

show abstract

Improving the replicability of neuroimaging findings by thresholding effect sizes instead of p‐values

Cited by 9 publications

References 32 publications

Functional neuroimaging in the era of Big Data and Open Science: A modern overview

Functional neuroimaging in the era of Big Data and Open Science: A modern overview

Long-lasting effects of insufficient sleep on neurocognitive development in early adolescence

Improving the replicability of neuroimaging findings by thresholding effect sizes instead of p‐values

Contact Info

Product

Resources

About