Evaluating methods of correcting for multiple comparisons implemented in SPM12 in social neuroscience fMRI studies: an example from moral psychology

Abstract

In fMRI research, the goal of correcting for multiple comparisons is to identify areas of activity that reflect true effects, and thus would be expected to replicate in future studies. Finding an appropriate balance between trying to minimize false positives (Type I error) while not being too stringent and omitting true effects (Type II error) can be challenging. Furthermore, the advantages and disadvantages of these types of errors may differ for different areas of study. In many areas of social neuroscience that involve complex processes and considerable individual differences, such as the study of moral judgment, effects are typically smaller and statistical power weaker, leading to the suggestion that less stringent corrections that allow for more sensitivity may be beneficial, but also result in more false positives. Using moral judgment fMRI data, we evaluated four commonly used methods for multiple comparison correction implemented in SPM12 by examining which method produced the most precise overlap with results from a meta-analysis of relevant studies and with results from nonparametric permutation analyses. We found that voxel-wise thresholding with family-wise error correction based on Random Field Theory provides a more precise overlap (i.e., without omitting too few regions or encompassing too many additional regions) than either clusterwise thresholding, Bonferroni correction, or false discovery rate correction methods. Evaluating Methods of Correcting for Multiple Comparisons Implemented in SPM12 in SocialNeuroscience fMRI Studies: An Example from Moral Psychology Correcting for multiple comparisons has been one of the most significant challenges in the statistical analysis of fMRI data . Because more than one hundred thousand voxels are compared simultaneously during analysis, the chances of Type I error are very high in the absence of any correction (Genovese, Lazar, & Nichols, 2002). In order to address this issue, researchers have developed various correction methods. For instance, Bonferroni's correction method, one of the traditional methods for multiple comparison correction, divides the nominal significance level (e.g., p < .05) by the number of tests being performed (Bland & Altman, 1995). Although Bonferroni correction produces good control of Type I error, it has the disadvantage of removing both false and true positives when applied to whole brain analyses. To address this issue, many researchers use a family-wise error (FWE) correction method based on Random Field Theory (RFT) (Nichols, 2012). Unlike the traditional Bonferroni method, which only accounts for the total number of comparisons, this method assumes that the error fields can be a lattice approximation to an underlying random field usually with a Gaussian distribution (Brett, Penny, & Kiebel, 2004;Eklund, Nichols, & Knutsson, 2016).Moreover, the false discovery rate (FDR) correction method was developed. This method is thought to be more sensitive and less likely to produce Type II error than FWE correction methods. U...

Using SPM 12’s Second-level Bayesian Inference Procedure for fMRI Analysis: Practical Guidelines for End Users

“…Furthermore, based on findings from such future studies, guidelines for analysis for end users might also need to be revised and amended. Second, researchers have developed alternative methods for inference and thresholding, such as Statistical Non-parametric Mapping (SnPM; Nichols, 2012), Threshold-free Cluster Enhancement (TFCE; Smith and Nichols, 2009), and 3dClustSim with autocorrelation function (Cox et al, 2017); previous research has demonstrated that the application of the aforementioned methods can effectively address current issues on fMRI analysis, such as inflated false positives (Nichols and Holmes, 2002;Smith and Nichols, 2009;Eklund et al, 2016;Han and Glenn, 2017). However, because those methods are not available as basic functions in SPM 12, we did not test the alternative methods with Bayesian inference due to the limited scope of the present study, introducing Bayesian inference to end users who are familiar with SPM 12's default settings.…”

Bayesian meta-analysis of fMRI image data

“…where V ovl is the number of voxels showing significant activity in both images, and V meta and V synth are the number of voxels showing significant activity in the result of our meta-analysis and that from NeuroSynth, respectively (Han & Glenn, 2018). This overlap index was employed in the present study because it does take into account both false positive voxels, voxels reported to be active in the meta-analysis result image but are not in fact active in the standard image, and false negative voxels, voxels reported to be inactive in the meta-analysis image but are in fact active in the standard image.…”

Using SPM 12’s Second-Level Bayesian Inference Procedure for fMRI Analysis: Practical Guidelines for End Users

“…Unfortunately, reports suggest that such may not be working well in practice. For example, an analysis by Han and Glenn’s (2017) shows Bonferroni correction and FDR are inappropriate for use because they are either too harsh or liberal in controlling for type I errors. Instead, they found that random field theory (RFT) familywise error correction (FWE)-applied voxel-wise thresholding struck a balance between the two methods, and they deemed it acceptable for fMRI data analysis in moral psychology.…”

“…Researchers can easily compute it without much statistical and computational knowledge. BF has several advantages when compared to P -values, which we will discuss in the next section, particularly in the cases of studies with small, underpowered samples, which are prevalent in social and cognitive neuroscientific studies ( Han and Glenn, 2017 ).…”