Post-selection Inference in Multiverse Analysis (PIMA): an inferential framework based on the sign flipping score test

Abstract

When analyzing data researchers make some decisions that are either arbitrary, based on subjective beliefs about the data generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused, and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that accounts for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e. pre-processing) and any generalized linear model; it allows testing the null hypothesis of a given predictor not being associated with the outcome, by merging information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate such that it allows researchers to claim that the null-hypothesis can be rejected for each specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Psychometrika SubmissionOctober 7, 2022 4Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real data set about coronavirus disease 2019 (COVID-19) vaccine hesitancy before and after the 2020 lockdown in Italy. We end with practical recommendations to consider when performing the proposed procedure.

Multiverse Meta-Analysis: Proposing an Exploratory Framework

“…The idea of specification curve analysis is quite simple; if there are several reasonable analyses to test the research question, and all are statistically valid and are not redundant with other analyses, we should run them all, summarize them in a curve plot (Figure 2.3), and evaluate the result across all of them (Simonsohn et al, 2020). This approach is the first attempt to draw inferential conclusions from previously exploratory multiverse results (Girardi et al, 2022) as it enables researchers to compare the result of the multiverse with a null distribution made by bootstrapping or permutations (Srivastava, 2018). However, it is not without limitations.…”

“…However, specification curve analysis cannot weigh them differently (Simonsohn et al, 2020). Additionally, it does not allow testing all possible specifications (Rauvola & Rudolph, 2023;Simonsohn et al, 2020), as well as it can only be run on simple cases related to the linear model (Girardi et al, 2022).…”

“…This can explain why the number of studies using multiverse analysis is still low (Rijnhart et al, 2021). Adding to that, those who are using multiverse analysis mostly stuck to descriptive reports rather than inferential conclusions (Girardi et al, 2022). This may be due to the scarcity of approaches to make inferential decisions from multiverse analysis.…”

“…We mentioned specification curve analysis as the first effort to draw inferential conclusions before, and we stated its drawbacks. Recently, Girardi et al (2022) have proposed the Post-selection…”