Theory and methods of the multiverse: an application for panel-based models

Abstract

Multiverse analysis involves systematically sampling a vast set of model specifications, known as a multiverse, to estimate the uncertainty surrounding the validity of a scientific claim. By fitting these specifications to a sample of observations, statistics are obtained as analytical results. Examining the variability of these statistics across different groups of model specifications helps to assess the robustness of the claim and gives insights into its underlying assumptions. However, the theoretical premises of multiverse analysis are often implicit and not universally agreed upon. To address this, a new formal categorisation of the analytical choices involved in modelling the set of specifications is proposed. This method of indexing the specification highlights that the sampling structure of the multiversal sample does not conform to a model of independent and identically distributed draws of specifications and that it can be modelled as an information network instead. Hamming’s distance is proposed as a measure of network distance, and, with an application to a panel dataset, it is shown how this approach enhances transparency in procedures and inferred claims and that it facilitates the check of implicit parametric assumptions. In the conclusions, the proposed theory of multiversal sampling is linked to the ongoing debate on how to weigh a multiverse, including the debate on the epistemic value of crowdsourced multiverses.

Characterisation and Calibration of Multiversal Models

“…. , q v } (1) where each q is a finite vector of q-states and v is the number of components elicited in the 'multiversal model' (Cantone and Tomaselli, 2024b). The q vectors of states can be understood with the metaphor of 'switches' in an engine: if in a generic multi-model, the analyst provides a list of blueprints for an engine, with a multiversal model Q the analyst provides a list of all the admissible states for all the q as a schema for generating blueprints.…”

“…Each unique combination of the q-states of q forms a specification j of the multiversal model. Cantone and Tomaselli (2024b) introduce a compact method for representing the junction of the identified specification ĵ, where specifications are represented as strings of symbols. Each q-state of q in the string represents a choice or a variant of the component, i.e.…”

“…a position for the switch. Symbols are relatively customary but the symbol '0' holds a special meaning: it indicates the concept of 'absent state' (the switch is turned off) of a decision, which Cantone and Tomaselli (2024b) interpret as a simpler state of the model. A typical example of 'absence state' is to consider the choice to adjust or not the estimation for a covariate, where q = 0 will be the symbol for absence of adjustment.…”

“…• A connection of the problem of identification of the plausibility and the normative implication of multiversal results with an extended formalism originally proposed in Cantone and Tomaselli (2024b), combining Network Theory and Information Theory. Section 3 addresses the technical challenges of the calibration of massive multi-model inference, with a focus on the estimation of the multiversal average estimate.…”

How to measure Interdisciplinary Research? A `systemic' design for the model of measurement

“…The origin of this crisis has been historically framed as a consequence of the bad incentives in peer-reviewed institutional research, where innovation and progress are overlooked over replication and robustness, with a consequential inflation of the frequency of false positive findings in published research (Nissen et al, 2016;Bartoš et al, 2024). Indeed, any reductionist paradigm of measurement can be 'hacked' just by omitting alternative choices in the model of measurement; in fact more robust forms of multi-model inference have been proposed as a new paradigm for inherently fickle research areas like social sciences (Steegen et al, 2016;Simonsohn et al, 2020;Cantone and Tomaselli, 2024).…”