Multiverse analysis for dynamic network models: Investigating the influence of plausible alternative modeling choices

Abstract

The analysis of time series data has become very popular in psychology. Specifying complex time series models involves many researchers' degrees of freedom, meaning that a wide range of plausible analysis strategies are possible. However, researchers typically perform and report only a single, preferred analysis while ignoring alternative assumptions and specifications that may lead to different conclusions. As a remedy, we propose multiverse analysis to investigate the robustness of time series network analysis to arbitrary, auxiliary modeling choices. We focus on group iterative multiple model estimation (GIMME), a highly data-driven modeling approach, and re-analyze two data sets (combined n=199) that were originally analyzed with GIMME. For each data set, we vary seven model parameters in a factorial design, resulting in 3,888 fitted models. We report the robustness of results at the group, subgroup, and individual levels and provide a web application to interactively explore our results. Group-level and, to a lesser extent, subgroup-level results were mostly stable across the multiverse with some differences between the two data sets. Individual-level estimates were more heterogeneous. Some modeling decisions (e.g., number of fit indices required for convergence) influenced results and conclusions more strongly. Overall, the robustness of GIMME to alternative modeling choices depends on the level of analysis. At the individual level, results may differ strongly even when changing the algorithm only slightly, which is highly relevant for applications such as clinical treatment selection and intervention. Multiverse analysis therefore is a valuable tool for checking the robustness of results from time series models.