Evaluation of Second-Level Inference in fMRI Analysis

Abstract

We investigate the impact of decisions in the second-level (i.e., over subjects) inferential process in functional magnetic resonance imaging on (1) the balance between false positives and false negatives and on (2) the data-analytical stability, both proxies for the reproducibility of results. Second-level analysis based on a mass univariate approach typically consists of 3 phases. First, one proceeds via a general linear model for a test image that consists of pooled information from different subjects. We evaluate models that take into account first-level (within-subjects) variability and models that do not take into account this variability. Second, one proceeds via inference based on parametrical assumptions or via permutation-based inference. Third, we evaluate 3 commonly used procedures to address the multiple testing problem: familywise error rate correction, False Discovery Rate (FDR) correction, and a two-step procedure with minimal cluster size. Based on a simulation study and real data we find that the two-step procedure with minimal cluster size results in most stable results, followed by the familywise error rate correction. The FDR results in most variable results, for both permutation-based inference and parametrical inference. Modeling the subject-specific variability yields a better balance between false positives and false negatives when using parametric inference.

Including Data Analytical Stability in Cluster-based Inference

“…It should be stressed that even with the current call to have more stringent control on the False Positives (Benjamin and Johnson, 2017), we would lack such advantages. While the concept of data analytical stability has recently been introduced mainly for the evaluation of methodological choices in fMRI data analysis (Roels et al, 2015(Roels et al, , 2016, it has previously been used to improve corrections for multiple testing (e.g. Gordon et al, 2009) also in fMRI (Durnez et al, 2014b).…”

“…Using the approach via testing, the concept was initially introduced in genetic association studies (e.g. Qiu et al, 2006;Gordon et al, 2007) but recently extended to fMRI (Durnez et al, 2014a;Roels et al, 2015Roels et al, , 2016. Roels et al (2016) demonstrated that for group studies, voxelwise FWE and FDR corrected analyses resulted in the same ROC curve and hence on average an equal trade-off between FP and FN (see also e.g.…”

“…Following this rationale, it is useful to consider the construction of voxelwise re-selection rates (Roels et al, 2016) to select stable activation regions.…”

“…In general, we determine 2 principled thresholds for cluster size at different levels α. As FWE correction was found to be less variable than FDR corrections (Qiu et al, 2006;Durnez et al, 2014b;Roels et al, 2016), we determine two FWE controlling thresholds for cluster size at cluster level: 1) a first relatively conservative threshold and 2) a more liberal threshold. Clusters above the conservative threshold are considered as strong evidence for activation and are retained.…”

Outcome Uncertainty and Brain Activity Aberrance in the Insula and Anterior Cingulate Cortex Are Associated with Dysfunctional Impulsivity in Borderline Personality Disorder

“…Their simulations also showed that it corresponded to FDR correction of q < 0.05 under common imaging parameters. A similar thresholding strategy was recently found to provide solid data reproducibility compared to both FDR and familywise error rate correction (Roels et al, 2016 ). Effect sizes (Cohen’s d ) were estimated in the hypothesized brain areas based on the mean Z -values for each participant in the significant clusters.…”