Analyzing Data of a Multilab Replication Project With Individual Participant Data Meta-Analysis

Aert, Robbie C. M. van

doi:10.1027/2151-2604/a000483

Cited by 6 publications

(12 citation statements)

References 71 publications

(116 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, if for instance age is theoretically expected to moderate an effect it should preferably be tested formally instead of being used in a multiverse analysis. van Aert (2022) demonstrated how this can be done by looking at the effect of age across labs in RRR9 (McCarthy et al, 2018) using IPD meta-analysis, finding a small positive interaction ( p = .038). Individual studies often do not have the power to detect moderating effects, which also affects multiverse analyses.…”

Section: Discussionmentioning

confidence: 99%

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Olsson-Collentine¹,

Aert²,

Bakker³

et al. 2023

Psychological Methods

View full text Add to dashboard Cite

Researcher degrees of freedom refer to arbitrary decisions in the execution and reporting of hypothesis-testing research that allow for many possible outcomes from a single study. Selective reporting of results ( p-hacking) from this "multiverse" of outcomes can inflate effect size estimates and false positive rates. We studied the effects of researcher degrees of freedom and selective reporting using empirical data from extensive multistudy projects in psychology (Registered Replication Reports) featuring 211 samples and 14 dependent variables. We used a counterfactual design to examine what biases could have emerged if the studies (and ensuing meta-analyses) had not been preregistered and could have been subjected to selective reporting based on the significance of the outcomes in the primary studies. Our results show the substantial variability in effect sizes that researcher degrees of freedom can create in relatively standard psychological studies, and how selective reporting of outcomes can alter conclusions and introduce bias in meta-analysis. Despite the typically thousands of outcomes appearing in the multiverses of the 294 included studies, only in about 30% of studies did significant effect sizes in the hypothesized direction emerge. We also observed that the effect of a particular researcher degree of freedom was inconsistent across replication studies using the same protocol, meaning multiverse analyses often fail to replicate across samples. We recommend hypothesis-testing researchers to preregister their preferred analysis and openly report multiverse analysis. We propose a descriptive index (underlying multiverse variability) that quantifies the robustness of results across alternative ways to analyze the data.

show abstract

Section: Discussionmentioning

confidence: 99%

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Olsson-Collentine¹,

Aert²,

Bakker³

et al. 2023

Psychological Methods

View full text Add to dashboard Cite

show abstract

“…Finally, we focused on effect sizes reported as mean differences between two independent groups. One-stage meta-analyses can model other types of effects (see van Aert, 2022) for which it might be interesLng to use the variance-covariance-matrix to explain effect heterogeneity as well.…”

Section: Implica#ons For Replica#on Research and The Intercept-slope ...mentioning

confidence: 99%

Whoever has will be given more? Exploring the Impact of Non-Linearity on Effect Heterogeneity in Psychological Research.

Fuenderich,

Beinhauer,

Renkewitz

2024

Preprint

View full text Add to dashboard Cite

Multi-lab projects are large scale collaborations between participating data collection sites that gather empirical evidence and (usually) analyze that evidence using meta-analyses. They are a valuable form of scientific collaboration, produce outstanding data sets and are a great resource for third-party researchers. Their data may be reanalyzed and used in research synthesis. Their repositories and code could provide guidance to future projects of this kind. But, while multi-labs are similar in their structure and aggregate their data using meta-analyses, they deploy a variety of different solutions regarding the storage structure in the repositories, the way the (analysis) code is structured and the file-formats they provide. Continuing this trend implies that anyone who wants to work with data from multiple of these projects, or combine their datasets, is faced with an ever-increasing complexity. Some of that complexity could be avoided. Here, we introduce MetaPipeX, a standardized framework to harmonize, document, analyze and multi-lab data. It features a pipeline conceptualization of the analysis and documentation process, an R-package that implements both and a Shiny App that allows users to explore and visualize these data sets. We introduce the framework by describing its components and applying it to a practical example. Engaging with this form of collaboration and integrating it further into research practice will certainly be beneficial to quantitative sciences and we hope the framework provides a structure and tools to reduce effort for anyone who creates, re-uses, harmonizes or learns about multi-lab replication projects.

show abstract

“…In the fifth and final article, Robbie van Aert (2022) discusses the value and benefits of conducting an individual participants meta-analysis when analyzing multiple replications of studies conducted in different labs that have become prominent in the many labs project (Ebersole et al, 2016). He points out to the weaknesses of meta-analytical approaches to aggregate the results of these studies, most notably the lower statistical power when analyzing moderator effects (in the form of differences across studies) and – more importantly – the danger of aggregation biases, that is, falsely concluding differences between studies to differences between individuals.…”

mentioning

confidence: 99%

Hotspots in Psychology – 2022 Edition

Steinmetz

Wedderhoff

Bošnjak

2022

Zeitschrift für Psychologie

View full text Add to dashboard Cite

Abstract. This editorial introduces the five articles included in the sixth “Hotspots in Psychology” of the Zeitschrift für Psychologie. With the new edition, the format enlarges it focus beyond meta-analyses and systematic reviews to new developments such as big data in psychology. The included five articles span a diverse array of topics, that is, the application of individual participants meta-analyses as a way to replicate studies, the role of the degree of anthropomorphism (“human-likeness”) in human-robot interactions, the challenge of multiple dependent effect sizes when conducting a meta-analytical structural equation model, the value of using log data of online platform as a way to predict learning outcomes, and the utility of a block-wise fit evaluation in structural equation models with many longitudinally measured variables. To promote the open science philosophy, the papers present supplemental material that can be accessed via the repository PsychArchives ( www.psycharchives.org ).

show abstract

Analyzing Data of a Multilab Replication Project With Individual Participant Data Meta-Analysis

Cited by 6 publications

References 71 publications

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Meta-analyzing the multiverse: A peek under the hood of selective reporting.

Whoever has will be given more? Exploring the Impact of Non-Linearity on Effect Heterogeneity in Psychological Research.

Hotspots in Psychology – 2022 Edition

Contact Info

Product

Resources

About