Improving practices and inferences in developmental cognitive neuroscience

Abstract

The past decade has seen growing concern about research practices in cognitive neuroscience, and psychology more broadly, that shake our confidence in many inferences in these fields. We consider how these issues affect developmental cognitive neuroscience, with the goal of progressing our field to support strong and defensible inferences from our neurobiological data. This manuscript focuses on the importance of distinguishing between confirmatory versus exploratory data analysis approaches in developmental cognitive neuroscience. Regarding confirmatory research, we discuss problems with analytic flexibility, appropriately instantiating hypotheses, and controlling the error rate given how we threshold data and correct for multiple comparisons. To counterbalance these concerns with confirmatory analyses, we present two complementary strategies. First, we discuss the advantages of working within an exploratory analysis framework, including estimating and reporting effect sizes, using parcellations, and conducting specification curve analyses. Second, we summarize defensible approaches for null hypothesis significance testing in confirmatory analyses, focusing on transparent and reproducible practices in our field. Specific recommendations are given, and templates, scripts, or other resources are hyperlinked, whenever possible.

How to capture developmental brain dynamics: gaps and solutions

“…Although large-scale studies seem vital to further advance our research field, researchers at the workshop also argued that these need to be complemented by exploratory studies, which are generally conducted in smaller samples and can target specific parts of the multilevel/function developmental framework. Exploratory research aims at building a knowledge base and generation of novel hypotheses 52 . Especially for a relatively young field of research such as developmental cognitive neuroscience, exploratory studies are essential, and allow for subsequent theory construction.…”

A Researcher’s Guide to the Measurement and Modeling of Puberty in the ABCD Study® at Baseline

“…For example, conclusions as to whether early pubertal maturation affects adult height in females differ when maturation is defined in terms of menarche versus breast development ( 39 ). This enhanced analytic flexibility ( 40 ) can have untoward effects in null-hypothesis significance testing, some of which can be mitigated by preregistration ( 41 ). While not comprehensive, potential a priori justifications for variable selection are presented in the section on PUBERTAL MEASURES in the ABCD DATASET .…”

Age-related change in task-evoked amygdala—prefrontal circuitry: a multiverse approach with an accelerated longitudinal cohort aged 4-22 years

“…Indeed, most studies involving many analytical decision points could benefit from multiverse analyses. Such specification curves can help to examine the stability of findings in both exploratory and confirmatory research 58 .…”