A Traveler’s Guide to the Multiverse: Promises, Pitfalls, and a Framework for the Evaluation of Analytic Decisions

Abstract

Decisions made by researchers while analyzing data (e.g., how to measure variables, how to handle outliers) are sometimes arbitrary, without an objective justification for choosing one alternative over another. Multiverse-style methods (e.g., specification curve, vibration of effects) estimate an effect across an entire set of possible specifications to expose the impact of hidden degrees of freedom and/or obtain robust, less biased estimates of the effect of interest. However, if specifications are not truly arbitrary, multiverse-style analyses can produce misleading results, potentially hiding meaningful effects within a mass of poorly justified alternatives. So far, a key question has received scant attention: How does one decide whether alternatives are arbitrary? We offer a framework and conceptual tools for doing so. We discuss three kinds of a priori nonequivalence among alternatives—measurement nonequivalence, effect nonequivalence, and power/precision nonequivalence. The criteria we review lead to three decision scenarios: Type E decisions (principled equivalence), Type N decisions (principled nonequivalence), and Type U decisions (uncertainty). In uncertain scenarios, multiverse-style analysis should be conducted in a deliberately exploratory fashion. The framework is discussed with reference to published examples and illustrated with the help of a simulated data set. Our framework will help researchers reap the benefits of multiverse-style methods while avoiding their pitfalls.

Multiverse analyses in fear conditioning research

“…This may not be particularly surprising as different analytic strategies may not test exactly the same underlying hypothesis but may -intentionally or unintentionally -test different hypotheses. This is true for models with and without covariates (Del Giudice & Gangestad, 2021) but also for models using different numbers of trials. Including only the first 2 trials of a (delayed) extinction phase tests for fear recall, while including only the last two trials tests for end-point extinction learning successes and trial-by-trial analyses test for temporal dynamics during extinction learning.…”

“…To this end, the multiverse analyses can either be run as the major analysis or may be included as an additional supplementary analyses to inform on the robustness of a reported finding. Most importantly, we anticipate that an increase in multiverse-type of studies will guide and aid the development of formal theories (Del Giudice & Gangestad, 2021) through the accumulation of empirical evidence guiding their development which we anticipate to ultimately contribute to a more successful and faster translation of fear conditioning research to clinical applications.…”

Assessing the Robustness of Mediation Analysis Results Using Multiverse Analysis

“…The general goal of a multiverse analysis is to assess the robustness of the effect estimates against data analytical decisions (Simonsohn et al, 2020;Steegen et al, 2016). It helps to identify the most impactful decisions and thereby provides important information for the development of a more complete and precise research theory (Del Giudice & Gangestad, 2021;Steegen et al, 2016). Multiverse analyses may be applied in original studies or to assess the robustness of previously published results.…”

“…These alternative decisions should be consistent with the underlying theoretical framework, statistically valid, and not redundant with other decisions in the multiverse (Simonsohn et al, 2020). In other words, the alternative decisions should reflect the arbitrary RDFs, but not include QRPs (Del Giudice & Gangestad, 2021;Wicherts et al, 2016). In contrast with conventional sensitivity analyses based on alternative decisions selected by the researcher, the goal of a multiverse analysis is to identify all decision points and all reasonable alternative decisions.…”

Graphing common stats tables

“…(Simonsohn, Simmons, & Nelson, 2020) is closely related to multiverse analysis and includes an inferential method for assessing the overall evidence for the research hypothesis using under-the-null resampling (i.e., permutation tests). This is a valuable approach, but it does not reveal which decisions (if any) consistently impact the statistical significance of the result or magnitude of the effect (for related discussion see Del Giudice & Gangestad, 2021).…”