Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses

Abstract

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.

Subjective Evidence Evaluation Survey For Many-Analysts Studies

“…We contribute to the current literature about guidelines on many-analysts studies (Aczel et al, 2021) by offering concrete advice on how to analyze and interpret (part of) the data obtained in many-analysts projects. This, together with advancements on synthesizing objective outcome metrics across analyses based on the same data (e.g., Coretta et al, 2023;Kümpel & Hoffmann, 2022), can move the field beyond drawing conclusions based on (visual) inspection of the analysts' outcomes.…”

“…Although measuring the distribution of plausible effect sizes can provide important insights about the robustness of an empirical result (e.g., Coretta et al, 2023;Kümpel & Hoffmann, 2022), we argue that it is incomplete (see also, Mathur et al, 2023;Young & Holsteen, 2017). To reap the full benefits of involving multiple analysts, we should also examine the broader context in which analysts made their choices: their prior beliefs about 1 Alternative approaches for synthesizing outcomes in many-analysts projects (e.g., considering only the sign of the effect size; focusing on evidential measures such as p-values or Bayes factors) do not seem satisfactory, especially when quantifying the size of the effect is essential (see e.g., Mathur et al, 2023).…”

“…Silberzahn et al, 2018 or a standardized beta coefficient, e.g., Hoogeveen, Sarafoglou, Aczel, et al, 2023, but see Trübutschek et al, 2023. These effect size estimates are (visually) summarized to provide an overall impression of the results (but see Kümpel andHoffmann, 2022 andCoretta et al, 2023 for recently proposed alternative statistical approaches).…”

Accuracy in Parameter Estimation and Simulation Approaches for Sample Size Planning with Multiple Stimuli

“…Changes in publication practices and research design have also created new challenges in providing a sample size plan for a research study. While statistics courses often suggest that a specific research design leads to a specific statistical test, meta-science work has shown that given the same data and hypothesis, researchers can come up with multiple ways to analyze the data (Coretta et al, 2023;Silberzahn et al, 2018). Therefore, a single power analysis only corresponds to the specific analysis that the researcher expects to implement.…”

Reproducible research practices and transparency across linguistics

“…are also prevalent in our field (Casillas, 2021;Kirby & Sonderegger, 2018;Laurinavichyute et al, 2022;Roettger, 2019;Vasishth et al, 2018). Recent meta-scientific assessments have investigated some aspects of research practices in language research, including the prevalence of direct replications (Kobrock & Roettger, 2023;Marsden, Morgan-Short, et al, 2018) and analytical flexibility (Coretta et al, 2023). Assessments of second language research Plonsky et al, 2015) reported limited sharing of materials (4-17%) and data (15%), and bilingualism researchers suggest poor availability of data, analysis, and materials in their subfield (Bolibaugh et al, 2021).…”