Data‐driven region‐of‐interest selection without inflating Type I error rate

Abstract

In ERP and other large multidimensional neuroscience data sets, researchers often select regions of interest (ROIs) for analysis. The method of ROI selection can critically affect the conclusions of a study by causing the researcher to miss effects in the data or to detect spurious effects. In practice, to avoid inflating Type I error rate (i.e., false positives), ROIs are often based on a priori hypotheses or independent information. However, this can be insensitive to experiment-specific variations in effect location (e.g., latency shifts) reducing power to detect effects. Data-driven ROI selection, in contrast, is nonindependent and uses the data under analysis to determine ROI positions. Therefore, it has potential to select ROIs based on experiment-specific information and increase power for detecting effects. However, data-driven methods have been criticized because they can substantially inflate Type I error rate. Here, we demonstrate, using simulations of simple ERP experiments, that data-driven ROI selection can indeed be more powerful than a priori hypotheses or independent information. Furthermore, we show that data-driven ROI selection using the aggregate grand average from trials (AGAT), despite being based on the data at hand, can be safely used for ROI selection under many circumstances. However, when there is a noise difference between conditions, using the AGAT can inflate Type I error and should be avoided. We identify critical assumptions for use of the AGAT and provide a basis for researchers to use, and reviewers to assess, data-driven methods of ROI localization in ERP and other studies.

Event-related brain potentials reveal age-related changes in parafoveal-foveal integration during sentence processing

“…Time-window ROI selection of the N400 was conducted based on inspection of the aggregate grand average across all trials (see Brooks et al, 2017), a waveform computed by aggregating all trials across all subjects and conditions, which allows for visualization of overall ERP component morphology independently from condition differences in ERP components, thus reducing the risk for increased Type-I error. An a priori ROI window length (200ms) was used based on prior work defining N400 component activity as maximal between 300–500ms (cf.…”

Visuospatial sequence learning on the serial reaction time task modulates the P1 event‐related potential

“…Latency ranges for the ERP components were derived using a data-driven procedure outlined by Brooks, Zoumpoulaki, and Bowman (2017). This procedure has been shown to reduce Type I error rates with respect to identifying regions of interest on ERP waveforms.…”

I Tried a Bunch of Things: The Dangers of Unexpected Overfitting in Classification

“…Furthermore, while there has been a lot of discussion of problems of circularity and inflated effects in neuroscience analyses (e.g. Kriegeskorte, Simmons, Bellgowan & Baker 2009;Vul, Harris, Winkielman & Pashler 2009;Eklund, Nichols, Anderson & Knutsson 2015;Brooks, Zoumpoulaki & Bowman, 2017), machine learning algorithms are so effective that they provide dangers above and beyond those that have been discussed. Optimization of hyper-parameters is a common practice in the machine learning literature (Bouthillier & Varoquaux 2020) and it is difficult to determine how the data were treated during the optimization process.…”

“…At the end of this paper, we will discuss several preventative solutions, including blind analysis. Unlike physics, while related issues have been discussed in the literature (Kriegeskorte et al 2009;Brooks et al 2017), the neuroscience field has not yet fully responded to the dangers of over-hyping when complex analyses are used, which increases the potential of false findings and presents a major barrier to the replicability of the literature.…”