A formal framework for generalized reporting methods in parametric settings

Abstract

Effect size measures and visualization techniques aimed at maximizing the interpretability and comparability of results from statistical models have long been of great importance and are recently again receiving increased attention in the literature. However, since the methods proposed in this context originate from a wide variety of disciplines and are more often than not practically motivated, they lack a common theoretical framework and many quantities are narrowly or heuristically defined. In this work, we put forward a common mathematical setting for effect size measures and visualization techniques aimed at the results of parametric regression and define a formal framework for the consistent derivation of both existing and new variants of such quantities. Throughout the presented theory, we utilize probability measures to derive weighted means over areas of interest. While we take a Bayesian approach to quantifying uncertainty in order to derive consistent results for every defined quantity, all proposed methods apply to the results of both frequentist and Bayesian inference. We apply selected specifications derived from the proposed framework to data from a clinical trial and a multi-analyst study to illustrate its versatility and relevance.

Subjective Evidence Evaluation Survey For Many-Analysts Studies

“…We contribute to the current literature about guidelines on many-analysts studies (Aczel et al, 2021) by offering concrete advice on how to analyze and interpret (part of) the data obtained in many-analysts projects. This, together with advancements on synthesizing objective outcome metrics across analyses based on the same data (e.g., Coretta et al, 2023;Kümpel & Hoffmann, 2022), can move the field beyond drawing conclusions based on (visual) inspection of the analysts' outcomes.…”

“…Although measuring the distribution of plausible effect sizes can provide important insights about the robustness of an empirical result (e.g., Coretta et al, 2023;Kümpel & Hoffmann, 2022), we argue that it is incomplete (see also, Mathur et al, 2023;Young & Holsteen, 2017). To reap the full benefits of involving multiple analysts, we should also examine the broader context in which analysts made their choices: their prior beliefs about 1 Alternative approaches for synthesizing outcomes in many-analysts projects (e.g., considering only the sign of the effect size; focusing on evidential measures such as p-values or Bayes factors) do not seem satisfactory, especially when quantifying the size of the effect is essential (see e.g., Mathur et al, 2023).…”

“…Silberzahn et al, 2018 or a standardized beta coefficient, e.g., Hoogeveen, Sarafoglou, Aczel, et al, 2023, but see Trübutschek et al, 2023. These effect size estimates are (visually) summarized to provide an overall impression of the results (but see Kümpel andHoffmann, 2022 andCoretta et al, 2023 for recently proposed alternative statistical approaches).…”