Handling Multiplicity in Neuroimaging through Bayesian Lenses with Multilevel Modeling

Abstract

In neuroimaging, the multiplicity issue may sneak into data analysis through several channels, affecting expected false positive rates (FPRs; type I errors) in diverse ways. One widely recognized aspect of multiplicity, multiple testing, occurs when the investigator fits a separate model for each voxel in the brain. However, multiplicity also occurs when the investigator conducts multiple comparisons within a model, tests two tails of a t-test separately when prior information is unavailable about the directionality, and branches in the analytic pipelines. The current practice of handling multiple testing through controlling the overall FPR in neuroimaging under the null hypothesis significance testing (NHST) paradigm excessively penalizes the statistical power with inflated type II errors. More fundamentally, the adoption of dichotomous decisions through sharp thresholding under NHST may not be appropriate when the null hypothesis itself is not pragmatically relevant because the effect of interest takes a continuum instead of discrete values and is not expected to be null in most brain regions.When the noise inundates the signal, two different types of error are more relevant than the concept of FPR: incorrect sign (type S) and incorrect magnitude (type M).In light of these considerations, we introduce a different strategy using Bayesian hierarchical modeling (BHM) to achieve two goals: 1) improving modeling efficiency via one integrative (instead of many separate) model and dissolving the multiple testing issue, and 2) turning the focus of conventional NHST on FPR into quality control by calibrating type S errors while maintaining a reasonable level of inference efficiency. The performance and validity of this approach are demonstrated through an application at the region of interest (ROI) level, with all the regions on an equal footing: unlike the current approaches under NHST, small regions are not disadvantaged simply because of their physical size. In addition, compared to the massively univariate approach, BHM may simultaneously achieve increased spatial specificity and inference efficiency. The benefits of BHM are illustrated in model performance and quality checking using an experimental dataset. In addition, BHM offers an alternative, confirmatory, or complementary approach to the conventional whole brain analysis under NHST, and promotes results reporting in totality and transparency. The methodology also avoids sharp and arbitrary thresholding in the p-value funnel to which the multidimensional data are reduced. The modeling approach with its auxiliary tools will be available as part of the AFNI suite for general use.

A tail of two sides: Artificially doubled false positive rates in neuroimaging due to the sidedness choice with t‐tests

“…As one of the four multiplicity issues typically involved in neuroimaging (Chen et al, ), the sidedness issue is not a matter of interpretation about effect directionality but simply a black‐and‐white, quantitative fact about FPR controllability. One way to observe this is to compare plots of relative areas of null hypothesis rejection for a given threshold, as demonstrated in Figure , where the significance level α = 0.1 is applied to an example of Student's t 20 ‐distribution.…”

“…For instance, a decision at the regional level can be adopted in an ROI‐based analysis approach through Bayesian multilevel modeling (Chen et al, ).…”

“…The commonality of this situation is highlighted by an anecdote: during a recent manuscript review process of ours, the statement, “testing both tails in tandem … is widely used” (Chen et al, ) evoked an unexpectedly dismissive denial from a reviewer: “Really? I disagree.…”

Untangling the Relatedness among Correlations, Part III: Inter-Subject Correlation Analysis through Bayesian Multilevel Modeling for Naturalistic Scanning

“…(5) Model inefficiency. The methods of correction for multiplicity tend to be over-penalizing (Chen et al, 2019a), and dichotomous decisions under NHST through thresholding are controversial in general (McShane et al, 2017;Amrhein and Greenland, 2017) and equally problematic in neuroimaging as well (Chen et al, 2019a). For instance, the popular practice of only reporting "statistically significant" results in neuroimaging not only wastes data information, but also distorts the full results as well as perpetuates the reproducibility crisis because of the fact that the difference between a "significant" result and a "non-significant" one is not necessarily significant (Cox et al, 1977).…”

“…To address those limitations, here we propose a Bayesian multilevel (BML) framework that integrates all the spatial elements (i.e., regions of interest) into one model. Such a framework has been applied to typical task-related FMRI experiments (Chen et al, 2019a;Xiao et al, 2019) as well as matrix-based data analysis (Chen et al, 2019b;Yin et al, 2019). A dataset of naturalist scanning is utilized to illustrate the modeling approach and to showcase the modeling capability, flexibility and advantages in reporting results.…”