Consensus-based guidance for conducting and reporting multi-analyst studies

Aczél, Balázs; Szászi, Barnabás; Nilsonne, Gustav; Akker, Olmo R. van den; Albers, Casper J.; Assen, Marcel ALM van; Bastiaansen, Jojanneke A.; Benjamin, Daniel J.; Boehm, Udo; Botvinik‐Nezer, Rotem; Bringmann, Laura F.; Busch, Niko A.; Caruyer, Emmanuel; Cataldo, Andrea M.; Cowan, Nelson; Delios, Andrew; Dongen, Noah van; Donkin, Chris; Doorn, Johnny van; Dreber, Anna; Dutilh, Gilles; Egan, Gary F.; Gernsbacher, Morton Ann; Hoekstra, Rink; Hoffmann, Sabine; Holzmeister, Felix; Huber, Jürgen; Johannesson, Magnus; Jonas, Kai J.; Kindel, Alexander; Kirchler, Michael; Kunkels, Yoram Kevin; Lindsay, D. Stephen; Mangin, Jean François; Matzke, Dora; Munafò, Marcus R.; Newell, Ben R.; Nosek, Brian A.; Poldrack, Russell A.; Ravenzwaaij, Don van; Rieskamp, Jörg; Salganik, Matthew J.; Sarafoglou, Alexandra; Schönberg, Tom; Schweinsberg, Martin; Shanks, David R.; Silberzahn, Raphael; Simons, Daniel J.; Spellman, Barbara A.; St-Jean, Samuel; Starns, Jeffrey J.; Uhlmann, Eric Luis; Wicherts, Jelte M.; Wagenmakers, Eric‐Jan

doi:10.7554/elife.72185

Cited by 44 publications

(48 citation statements)

References 43 publications

(73 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 Inviting independent analysts from various backgrounds including but not restricted to religious studies attenuates this potential concern. Moreover, in addition to quantifying variability, with a sufficiently large number of analysis teams one can also investigate factors that might explain observed variability, such as those related to theoretical or methodological expertise and prior beliefs (Aczel et al, 2021). 2…”

Section: A Many-analysts Approachmentioning

confidence: 99%

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

Hoogeveen¹,

Sarafoglou²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N=10,535 participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported beta = 0.120). For the second research question, this was the case for 65% of the teams (median reported beta = 0.039). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates.

show abstract

Section: A Many-analysts Approachmentioning

confidence: 99%

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

Hoogeveen¹,

Sarafoglou²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To additionally increase the integrity and the reproducibility of these analyses the analysts can also be “blinded” regarding the initial hypothesis of the research. Recently, the guideline for multi-analyst research emerged 45 wrapping up the techniques of strengthening the robustness of scientific results. As the different analysis pipelines used by research groups could possibly lead to different conclusions, authors encourage independent researchers to perform an analysis on the same data, inferring the robust result where results from the different researchers converge.…”

Section: Discussionmentioning

confidence: 99%

The thresholding problem and variability in the EEG graph network parameters

Adamovich

Zakharov

Tabueva

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Graph thresholding is a frequently used practice of eliminating the weak connections in brain functional connectivity graphs. The main aim of the procedure is to delete the spurious connections in the data. However, the choice of the threshold is arbitrary, and the effect of the threshold choice is not fully understood. Here we present the description of the changes in the global measures of a functional connectivity graph depending on the different proportional thresholds based on the 146 resting-state EEG recordings. The dynamics is presented in five different synchronization measures (wPLI, ImCoh, Coherence, ciPLV, PPC) in sensors and source spaces. The analysis shows significant changes in the graph’s global connectivity measures as a function of the chosen threshold which may influence the outcome of the study. The choice of the threshold could lead to different study conclusions; thus it is necessary to improve the reasoning behind the choice of the different analytic options and consider the adoption of different analytic approaches. We also proposed some ways of improving the procedure of thresholding in functional connectivity research.

show abstract

“…An extreme example is that by Bastiaansen et al (2020) where the same data were analyzed by various researchers, resulting in heterogeneous results. The impacts of such legitimate researcher degrees of freedom on the generalizability of findings across research groups can be studied with multi-analysts studies (Aczel et al, 2021;Bastiaansen et al, 2020;Silberzahn et al, 2018) and multiverse analyses (Dragicevic et al, 2019;Steegen et al, 2016).…”

Section: Legitimate Researcher Degrees Of Freedom In Decisions About ...mentioning

confidence: 99%

Generalizability Crisis Meets Heterogeneity Revolution: Determining Under Which Boundary Conditions Findings Replicate and Generalize

Moeller¹,

Dietrich²,

Neubauer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Intensive longitudinal studies typically examine phenomena that vary across time, individuals, contexts, and other boundary conditions. This poses challenges to the conceptualization and identification of replicability and generalizability, which refer to the invariance of research findings across samples and contexts as crucial criteria for trustworthiness. Some of these challenges are specific to intensive longitudinal studies, others are similarly relevant for the work with other complex datasets that contain multilayered sources of variation (individuals nested in different types of activities or organizations, regions, countries, etc.)This article opens with discussing the reasons why research findings may fail to replicate. We then analyze reasons why research findings may falsely appear to be non-replicable when in fact they were as such replicable, but lacked generalizability due to heterogeneity between samples, subgroups, individuals, time points, and contexts. Following that, we propose conceptual and methodological approaches to better disentangle non-replicability from non-generalizability and to better understand the exact causes of either problem. In particular, we apply Lakatos’s proposition to examine not only whether but under what boundary conditions a theory is a useful description of the world, to the question whether and under which conditions a research finding is replicable and generalizable. Not only will that contribute to a more systematic understanding of and research on replicability and generalizability in longitudinal studies and beyond, but it will also be a contribution to what has been called the heterogeneity revolution (Bryan et al., 2021; Moeller, 2021).

show abstract

Consensus-based guidance for conducting and reporting multi-analyst studies

Cited by 44 publications

References 43 publications

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

The thresholding problem and variability in the EEG graph network parameters

Generalizability Crisis Meets Heterogeneity Revolution: Determining Under Which Boundary Conditions Findings Replicate and Generalize

Contact Info

Product

Resources

About