The case for preregistering all region of interest (ROI) analyses in neuroimaging research

Abstract

In neuroimaging studies, small sample sizes and the resultant reduced statistical power to detect effects that are not large, combined with inadequate analytic choices, concur to produce inflated or false‐positive findings. To mitigate these issues, researchers often restrict analyses to specific brain areas, using the region of interest (ROI) approach. Crucially, ROI analysis assumes the a priori justified definition of the target region. Nonetheless, reports often lack details about where in the timeline, ranging from study conception to the data analysis and interpretation of findings, were ROIs selected. Frequently, the rationale for ROI selection is vague or inadequately founded on the existing literature. These shortcomings have important implications for ROI‐based studies, augmenting the risk that observed effects are inflated or even false positives. Tools like preregistration and registered reports could address this problem, ensuring the validity of ROI‐based studies. The benefits could be enhanced by additional practices such as selection of ROIs using quantitative methods (i.e., meta‐analysis) and the sharing of whole‐brain unthresholded maps of effect size, as well as of binary ROIs, in publicly accessible repositories.

Brain activity during facial processing in autism spectrum disorder: an activation likelihood estimation (ALE) meta‐analysis of neuroimaging studies

“…Finally, it is important to underscore that our results are derived from whole‐brain analysis only. Conversely, significant amygdala findings are often only detected in studies relying on ROI analyses, given its small volume and the use of stringent type I error methods, like the frequently used cluster‐wise approach (Gentili, Cecchetti, Handjaras, Lettieri, & Cristea, 2020). The use of other, less conservative but still valid methods for type I error correction, like false discovery rate, voxel‐wise correction, equitable thresholding, and clustering (Cox, 2019), appear better suited to detect differences in amygdala activation.…”

Parsing variability in borderline personality disorder: a meta-analysis of neuroimaging studies

“…Moreover, analytic pipelines including pre-processing, choices in data analysis like the type of multiple comparison corrections employed most likely diverged between studies, with direct consequences on the threshold for identifying statistical findings [45][46][47][48] . The relationship between analytic choices and reporting statistically significant findings is particularly relevant for ALE meta-analyses, which exclusively rely on these results and cannot consider non-significant one 49 .…”

“…Solutions for clarifying similar issues in the neuroimaging literature include access to unthresholded maps, for example by posting them in public repositories, so as to allow studies with null or negative findings to be included in neuroimaging meta-analyses. Likewise, pre-registration of ROI analyses 49 , by guaranteeing these analyses were not contingent to non-significant whole-brain results, could support their inclusion in meta-analyses. The results of the leave-one-out analysis for the emotional processing subgroup similarly point to limited robustness.…”

“…Another limitation is related to the considerable number of studies that were excluded due to reporting only on ROI analyses (N = 23). Though the ROI approach is widely employed in neuroimaging studies, particularly in small sample studies, it is associated with increased risk of false positives or overestimation of effects 49 . Moreover, its use is discouraged in coordinate based meta-analysis 23,26 .…”

Assessing neuromodulation effects of theta burst stimulation to the prefrontal cortex using TMS-evoked potentials

“…Even with corrections for multiple comparisons, global scalp cluster-based approaches may inflate false-positive rates, although the inflation size is debated (Cox et al, 2017). Restriction of analyses within specific brain areas, as with the use of a pre-registered ROI, offers a better balance between sensitivity and the number of performed comparisons (Gentili et al, 2021). Alternatively, the use of a collapsed localiser approach (Luck and Gaspelin, 2017, Cohen, 2014), blind to group condition, to select the region of interest may provide an intermediary compromise between preregistered ROIs and exploratory data-driven channel selection.…”