Specification Analysis for Technology Use and Teenager Well-Being: Statistical Validity and a Bayesian Proposal

Abstract

A key issue in science is assessing robustness to data analysis choices, while avoiding selective reporting and providing valid inference. Specification Curve Analysis is a tool intended to prevent selective reporting. Alas, when used for inference it can create severe biases and false positives, due to wrongly adjusting for covariates, and mask important treatment effect heterogeneity. As our motivating application, it led an influential study to conclude there is no relevant association between technology use and teenager mental well‐being. We discuss these issues and propose a strategy for valid inference. Bayesian Specification Curve Analysis (BSCA) uses Bayesian Model Averaging to incorporate covariates and heterogeneous effects across treatments, outcomes and subpopulations. BSCA gives significantly different insights into teenager well‐being, revealing that the association with technology differs by device, gender and who assesses well‐being (teenagers or their parents).

Characterisation and Calibration of Multiversal Models

“…The correct statistical treatment of Janus effect coincides with the so-called Type-S Error, or TSE Gelman and Tuerlinckx (2000); Gelman and Carlin (2014); Lu et al (2019), that is the probability that, given the j specification of a two-tailed test such that the p < α, then the sign associated to the estimates is the opposite of the true value of the estimand. The emergence of conflicting results supports the position of authors who pushed for a modernisation of Multiverse Analysis, recognising the importance of evaluation of a calibration a posteriori of the multiverse by the application of a different weighting scheme or updating the prior through the nullification of the plausibility of some q (Muñoz and Young, 2018a;Slez, 2019;Del Giudice and Gangestad, 2021;Semken and Rossell, 2022;Cantone and Tomaselli, 2024b).…”

“…The multiversal methodology allows to the analyst to fit the available data on all the plausible combinations of the elicited alternative components. The terms 'multiversal model' and 'multiverse' are synonymous but to avoid linguistic confusion 1 , the intuition of Semken and Rossell (2022) of adopting a language typical of Bayesian analysis can be followed: the set of specifications can be considered as a setup of prior values (prior of the multiversal model) and the multiversal statistics can be treated as posterior values.…”

“…where for multiverse k = k Q , i.e., the adoption of Uniform weights. As reported by Semken and Rossell (2022), this implicit Uniform calibration is the most adopted in literature but there are arguments for rejecting Eq. 11 as a logically sufficient weighting scheme for the posterior distributions.…”

“…These happen 7 The authors claim for a null average effect not because no effects are triggered by a variation of gender parity, but because the effect is mediated by the stratification of the dependent variable: the multiversal findings are coherent with the more complex hypothesis that gender parity has significant positive effects on the reproductive propensity of older women but negative effects on the reproductive propensity in younger women. The argument about how the average effect may hide many effects across groups is also discussed in Semken and Rossell (2022) about the permutation test on the median in Simonsohn et al (2020). The critique regards the classical inefficiency of medians to properly catch a meaningful location in asymmetric bi-modal empirical frequencies, especially where the heterogeneity in the bi-modal distribution is explainable a posteriori by the mixture effect of a stratification variable.…”

“…The relevance of this difference is generally ignored in applied methods involving Multiverse, which are treated as extended multi-model inference. For example, Slez (2019) and Semken and Rossell (2022) propose Bayesian weighting schemes for the calibration of the multiversal inference, according to a typical fashion of techniques for model averaging that assumes the independence of specifications and their inferential errors (Draper, 1995). About this equivalence in assumptions, it is important to not ignore the contribution of Young (2019) and Del Giudice and Gangestad (2021) who predict that for misspecifications of the causal model in the design of the multiverse, the results will be even more severely biased than with canonical multi-models.…”