Assessing the Robustness of Mediation Analysis Results Using Multiverse Analysis

Abstract

There is an increasing awareness that replication should become common practice in empirical studies. However, study results might fail to replicate for various reasons. The robustness of published study results can be assessed using the relatively new multiverse-analysis methodology, in which the robustness of the effect estimates against data analytical decisions is assessed. However, the uptake of multiverse analysis in empirical studies remains low, which might be due to the scarcity of guidance available on performing multiverse analysis. Researchers might experience difficulties in identifying data analytical decisions and in summarizing the large number of effect estimates yielded by a multiverse analysis. These difficulties are amplified when applying multiverse analysis to assess the robustness of the effect estimates from a mediation analysis, as a mediation analysis involves more data analytical decisions than a bivariate analysis. The aim of this paper is to provide an overview and worked example of the use of multiverse analysis to assess the robustness of the effect estimates from a mediation analysis. We showed that the number of data analytical decisions in a mediation analysis is larger than in a bivariate analysis. By using a real-life data example from the Longitudinal Aging Study Amsterdam, we demonstrated the application of multiverse analysis to a mediation analysis. This included the use of specification curves to determine the impact of data analytical decisions on the magnitude and statistical significance of the direct, indirect, and total effect estimates. Although the multiverse analysis methodology is still relatively new and future research is needed to further advance this methodology, this paper shows that multiverse analysis is a useful method for the assessment of the robustness of the direct, indirect, and total effect estimates in a mediation analysis and thereby to inform replication studies.

Mediating effect of pubertal stages on the family environment and neurodevelopment: An open-data replication and multiverse analysis of an ABCD Study®

“…Similar to the multiverse mediation analyses in Rijnhart et al (2021) , in Aim 2, the mediation results across our 135 mediation model permutations are reported using specification curves ( Simonsohn et al, 2020 ). A specification curve is reported independently for the direct, indirect and total effect.…”

“…To draw inferences across the specification curve, we report several results. First, like Rijnhart et al (2021) , we report the frequency and direction of the effect across the multiverse compared to the effects in Aim 1. Second, we consider the proportion of effects from Aim 1 that overlap with the 95% CI in Aim 2 for each brain outcome.…”

“…Across their model permutations, they reported age-related associations in functional activation in the amygdala were relatively robust to model permutations but findings for amygdala-medial prefrontal cortex connectivity were inconsistent across model permutations. Finally, Rijnhart et al (2021) used a multiverse analysis on a mediation model to examine indirect, direct and total effects of models evaluating whether fat mass mediated the association between weight change and bone mineral density. Across their models, they observed effects that were consistent and in agreement with existing work, which suggested that association between weight change and bone mineral density was robust overall, across alternative analytic decisions in their sample.…”

“…Within the constraints of the ABCD study design, we consider the theoretically plausible sub-factors and individual scales that may be used in future research to evaluate the family environment, as well as the parent and youth reported variable of pubertal development. We report the results of the multiverse analysis of the mediation model using specification curves ( Rijnhart et al, 2021 ; Simonsohn et al, 2020 ) for the direct, indirect and total effects.…”

Post-selection Inference in Multiverse Analysis (PIMA): an inferential framework based on the sign flipping score test

“…The method has become increasingly popular since it has been developed, and it has been applied in several experimental contexts, ranging from cognitive development and risk perception (Mirman et al, 2021), assessment of parental behavior (Modecki et al, 2020), and memory task (Wessel et al, 2020). While parts of the applications remain confined to exploratory purposes with the scope to define brief guidelines for conducting a multiverse analysis (Dragicevic et al, 2019;Liu et al, 2020), other studies aim to stimulate interest in this method as a robustness assessment for mediation analysis (Rijnhart et al, 2021), or an exhaustive modelling approach (Frey et al, 2021). This research approach permits to exhibit the stability and robustness of discoveries, not just between different exclusion criteria or modifications of the variables, but between different decisions for all phases of the elaboration of the data.…”